Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
Benji York wrote at 2008-7-3 17:44 -0400: >On Thu, Jul 3, 2008 at 5:37 PM, Christian Theune <[EMAIL PROTECTED]> wrote: >> On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: >>> I'd like to 1) remove the layer tear-down mechanism entirely, and 2) >>> make (almost) all layers run in a subprocess. You are aware that layers can be nested? The implication of this is that a sublayer (run in a subprocess) either must start from scratch and reconstruct the fixture built in the superlayer (potentially expensive) or must access the resources inherited from the forking process. The latter (accessing resources inherited from the forking process) is very brittle. I had to give it up in a different context. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
On Sat, Jul 5, 2008 at 3:18 AM, Christian Theune <[EMAIL PROTECTED]> wrote: > We could use layers as a hint to create subprocesses, but should split > up layers if they are too large to fit the X/N rule (maybe with a margin > of a few percent to avoid splits for single or few tests). It probably wouldn't be too hard to automatically break "large" layers into several small layers. It's also not hard for people with very large layers that care about parallel execution time to break them up themselves. I'm not opposed to automatic layer segmentation (as long as it's implemented well), but also don't think it's all that important. -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
On Sat, 2008-07-05 at 09:18 +0200, Christian Theune wrote: > Hi, > > On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > > I'm working on making the zope.testing test runner run tests in > > parallelized subprocesses. The option will likely be spelled -j N, > > where N is the maximum number of processes. > > Getting back to the idea about parallelizing on a per-test base and not > per-layer: > > The ZODB currently runs only unit tests (which became a true layer in > zope.testing/trunk) but takes about XX minutes on one of my machines (4 > core XEON, 3.2 GHz). The actual numbers are: Ran 2816 tests with 0 failures and 0 errors in 14 minutes 34.292 seconds. real14m36.099s user3m44.740s sys 0m43.170s -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
Hi, On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. Getting back to the idea about parallelizing on a per-test base and not per-layer: The ZODB currently runs only unit tests (which became a true layer in zope.testing/trunk) but takes about XX minutes on one of my machines (4 core XEON, 3.2 GHz). I'd suggest that the general principle of splitting up the runs over multiple parallel processes should happen in a way that if you have X total tests and N parallel processes, we should have roughly X/N tests run in parallel. We could use layers as a hint to create subprocesses, but should split up layers if they are too large to fit the X/N rule (maybe with a margin of a few percent to avoid splits for single or few tests). Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
Hi Benji, I've read the whole thread to date but thought I'd reply here... Benji York wrote: I'm working on making the zope.testing test runner run tests in parallelized subprocesses. The option will likely be spelled -j N, where N is the maximum number of processes. Cool :-) But please defauult to 1 for backwards compatability... I'd like to 1) remove the layer tear-down mechanism entirely, and 2) make (almost) all layers run in a subprocess. -lots to both of these I'm afraid. I've used tear-downs extensively for everything from shutting database connections to aborting transactions to dumping DemoStorages. I'm sure I'm not the only one.. As for all layers in a sub-process, I worry that this would break existing tests in some kind of horrible nasty way... I want to do #1 because it would simplify the test runner code and no one seems to be using the functionality anyway. It also appears from reading the code that any tests run in a subprocess (and most are) will never exercise the tear-down mechanism anyway. So I guess we're not currently running tests in a sub-process? My take on the pre-refactor testrunner was that a sub-process was only used when the testrunner was testing itself? #2 will add some process start-up overhead, but it'll only be one more process than is already started (and any reasonably-sized test corpus already starts several processes for each test run). The one exception is for running with -D or with a pdb.set_trace() embedded in the code under test. For that case we need a switch to say "don't start any subprocesses at all", I suspect that will be spelled -j0. Right, I use this a lot. I guess -j0 should be the default for backwards compatability? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
Hello Benji, +1 for keeping the default as no subprocess and keeping the teardown. The others already said the reasons. -- Best regards, Adam GROSZERmailto:[EMAIL PROTECTED] -- Quote of the day: It is a great mistake to suppose that God is only, or even chiefly, concerned with religion. - William Temple, Archbishop of Canterbury ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
On Thu, Jul 03, 2008 at 05:22:11PM -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. That's wonderful news! > I have it basically working, but have noticed a couple odd corners of > the test runner that I'd like to clean up. They may be controversial, > so I'll ask about them here first. > > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. -1 in general. +1 if you do that only for -j N where N > 1. Running all the tests in a single process has the following benefits: * test coverage analysis produces results that are correct (well, correct often enough -- but it has no chance at all when the test runner forks a subprocess) * import pdb; pdb.set_trace() works > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. That's news to me. A while ago I went through Zope 3 trunk (it was pre-eggsplosion IIRC) and made sure all test layers defined in it supported teardown. Granted, FunctionalTestLayer() has allow_teardown=False as the default, for two reasons: * backwards compatibility: in olden days functional test layers didn't support teardown * paranoia: it is in general impossible to determine whether calling CleanUp().cleanUp() will correctly clear all the global state (someone could easily write a custom ZCML directive that changed a global variable and forget to register a CleanUp hook), so disallowing teardowns were the conservative safe choice. It is entirely my fault that I haven't evangelized the allow_teardown=True option for creating new test layers. > It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. I guess that's fine for process state, but not so fine for external state (temporary files etc.). Hey, this might explain why SchoolTool's tests tend to fill up my buildbot's /tmp without cleaning up after themselves! I'll have to investigate some day. > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. If that case needs to be supported anyway, what's the advantage of spawning exactly one subprocess when you run it with -j 1? I would also question whether pdb-unfriendly non-performance-enhancing option should be the default. > For motivation, some speed comparisons: running a particular test suite > with 3876 tests (mostly doctests, and mostly functional) without the > patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 > minutes and 22 seconds (give or take) on a dual-core box with 3 > simultaneous subprocesses. I know; for large test suites (by "large" I mean 40 minutes) I've been using an ugly hack (--odd/--even test filtering) that lets me use both CPUs if I manually run two instances of the test runner in two xterms in parallel. Regards, Marius Gedminas -- "Wipe Info uses hexadecimal values to wipe files. This provides more security than wiping with decimal values." -- Norton SystemWorks 2002 Manual signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
On Thu, Jul 3, 2008 at 5:37 PM, Christian Theune <[EMAIL PROTECTED]> wrote: > On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: >> I'd like to 1) remove the layer tear-down mechanism entirely, and 2) >> make (almost) all layers run in a subprocess. >> >> I want to do #1 because it would simplify the test runner code and no >> one seems to be using the functionality anyway. It also appears from >> reading the code that any tests run in a subprocess (and most are) will >> never exercise the tear-down mechanism anyway. > > +1 in general but -1 on removing the tear down functionality. We use it > to destroy external databases that where generated for setup. Ah! Good point. >> #2 will add some process start-up overhead, but it'll only be one more >> process than is already started (and any reasonably-sized test corpus >> already starts several processes for each test run). The one exception >> is for running with -D or with a pdb.set_trace() embedded in the code >> under test. For that case we need a switch to say "don't start any >> subprocesses at all", I suspect that will be spelled -j0. > > +1 as well. I'm actually wondering whether we might be able to control > the pdb through a sub-process. I don't think it'd be that hard, in general, but the current design of using stdout and stderr for IPC communication channels is a hindrance. >> For motivation, some speed comparisons: running a particular test suite >> with 3876 tests (mostly doctests, and mostly functional) without the >> patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 >> minutes and 22 seconds (give or take) on a dual-core box with 3 >> simultaneous subprocesses. > > Yay! I have an 8 core machine that I can't wait to try it on. ;) -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
AW: [Zope-dev] Test runner: layers, subprocesses, and tear down
Hi Benji > Betreff: [Zope-dev] Test runner: layers, subprocesses, and tear down [... ] > #2 will add some process start-up overhead, but it'll only be > one more process than is already started (and any > reasonably-sized test corpus already starts several processes > for each test run). The one exception is for running with -D > or with a pdb.set_trace() embedded in the code under test. > For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. That's a very important point. I often use pdb if I write tests. > For motivation, some speed comparisons: running a particular > test suite with 3876 tests (mostly doctests, and mostly > functional) without the patch takes 6 minutes, 42 seconds; my > branch runs the same tests in 3 minutes and 22 seconds (give > or take) on a dual-core box with 3 simultaneous subprocesses. Yeah, great! Regards Roger Ineichen > Benji York > Senior Software Engineer > Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Test runner: layers, subprocesses, and tear down
On Thu, 2008-07-03 at 17:22 -0400, Benji York wrote: > I'm working on making the zope.testing test runner run tests in > parallelized subprocesses. The option will likely be spelled -j N, > where N is the maximum number of processes. > > I have it basically working, but have noticed a couple odd corners of > the test runner that I'd like to clean up. They may be controversial, > so I'll ask about them here first. > > I'd like to 1) remove the layer tear-down mechanism entirely, and 2) > make (almost) all layers run in a subprocess. > > I want to do #1 because it would simplify the test runner code and no > one seems to be using the functionality anyway. It also appears from > reading the code that any tests run in a subprocess (and most are) will > never exercise the tear-down mechanism anyway. +1 in general but -1 on removing the tear down functionality. We use it to destroy external databases that where generated for setup. > #2 will add some process start-up overhead, but it'll only be one more > process than is already started (and any reasonably-sized test corpus > already starts several processes for each test run). The one exception > is for running with -D or with a pdb.set_trace() embedded in the code > under test. For that case we need a switch to say "don't start any > subprocesses at all", I suspect that will be spelled -j0. +1 as well. I'm actually wondering whether we might be able to control the pdb through a sub-process. > For motivation, some speed comparisons: running a particular test suite > with 3876 tests (mostly doctests, and mostly functional) without the > patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 > minutes and 22 seconds (give or take) on a dual-core box with 3 > simultaneous subprocesses. Yay! -- Christian Theune · [EMAIL PROTECTED] gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Test runner: layers, subprocesses, and tear down
I'm working on making the zope.testing test runner run tests in parallelized subprocesses. The option will likely be spelled -j N, where N is the maximum number of processes. I have it basically working, but have noticed a couple odd corners of the test runner that I'd like to clean up. They may be controversial, so I'll ask about them here first. I'd like to 1) remove the layer tear-down mechanism entirely, and 2) make (almost) all layers run in a subprocess. I want to do #1 because it would simplify the test runner code and no one seems to be using the functionality anyway. It also appears from reading the code that any tests run in a subprocess (and most are) will never exercise the tear-down mechanism anyway. #2 will add some process start-up overhead, but it'll only be one more process than is already started (and any reasonably-sized test corpus already starts several processes for each test run). The one exception is for running with -D or with a pdb.set_trace() embedded in the code under test. For that case we need a switch to say "don't start any subprocesses at all", I suspect that will be spelled -j0. For motivation, some speed comparisons: running a particular test suite with 3876 tests (mostly doctests, and mostly functional) without the patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3 minutes and 22 seconds (give or take) on a dual-core box with 3 simultaneous subprocesses. -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
