Re: [Zope] Testing products: Testing all in one batch
Paul Winkler wrote: I have a python script that builds a big command line to test.py of the form: bin/zopectl test Products/Product1|Products/Products2|etc ...it's not pretty, but it does work, and lets you exclude geb0rken products like Archetypes from test runs... Aha, that's much like what I'm doing except that yours runs in one go and thus exposes tests that don't clean up after themselves. Which is a Good Thing if I can just find and fix all the nasty things... No, no, we have several batches above, where each batch contains only products that don't screw up each others' tests. There were some products, mainly from Plone, which we just had to remove from the test run altogether, because THEY SUCK! ;-) Irrelevant to this discussion, but A) it's hardly ancient, But CMF 1.6 is on the horizon now ;-) and B) upgrading to 1.5.x has been postponed indefinitely as it breaks a lot of old things (some mine, some third-party) and I have not had any time to devote a day or three to resolving the incompatibilities. Ah, the compatability trap :-S - Some of my own Product tests run fine in isolation but break when run alongside other installed product tests. Then either your tests or the other tests are leaving turds... Indeed. I wish it was easier to diagnose and resolve problems like this. Yup, I know of no other way that goign through case-by-case... The problem is that tests seem to be run in alphabetical order, I don't know of a way to force another order, go and get inside unittest.py and have a play... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
Paul Winkler wrote: Has anybody set up a batch job to test all installed Products in a zope instance? Yes ;-) Seems like the only *useful* way I can get it to run is with a loop that does bin/zopectl test once for each subdirectory of Products that I actually care about, taking care to ignore problematic third-party products. Which is OK I guess, but I'd prefer not to have to skip anything. I have a python script that builds a big command line to test.py of the form: bin/zopectl test Products/Product1|Products/Products2|etc ...it's not pretty, but it does work, and lets you exclude geb0rken products like Archetypes from test runs... - Some otherwise excellent third-party products have tests with dependencies that the product itself does not have, and which I do not want to install. (I will contact the author(s) privately.) Yep, I exclude them too ;-) - Some third-party products (e.g. CMF) ship with tests that just plain fail. Really? That's not been the case for a long time. When trying to test a Plohn setup, I found some of the Plohn tests were causing CMF tests to fail if the CMF tests were run in the same batch, 'cos they no doubt leave lots of crap lying around :-( If I install CMF 1.4.8 and nothing else, and run ./bin/zopectl test, I get 1 error and 1 failure. Why are you using something so ancient? ;-) - Some of my own Product tests run fine in isolation but break when run alongside other installed product tests. Then either your tests or the other tests are leaving turds... - Does this jibe with others' experience? Yes. - Is there a better (or best) way to run all my Product tests? If you find it, let me know! :-) - *Should* we expect that ./bin/zopectl test will run all installed Products tests, and if so, what can we do as a community to get authors to fix their products? Beat them with sticks, embarass them in public, etc... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
On 4 Oct 2005, at 23:21, Paul Winkler wrote: Has anybody set up a batch job to test all installed Products in a zope instance? bin/zopectl test --dir Products/ from the instance home will test all products in that instance home. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
Jens Vagelpohl said: On 4 Oct 2005, at 23:21, Paul Winkler wrote: Has anybody set up a batch job to test all installed Products in a zope instance? bin/zopectl test --dir Products/ from the instance home will test all products in that instance home. Well, so will bin/zopectl test without the --dir argument, but I guess you didn't read the rest of my message ;-) Thanks anyway. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
On 5 Oct 2005, at 14:44, Paul Winkler wrote: bin/zopectl test --dir Products/ from the instance home will test all products in that instance home. Well, so will bin/zopectl test without the --dir argument, but I guess you didn't read the rest of my message ;-) I answered one of the questions you asked, that's all. The issue you are having with tests failing when you run them is separate. It's a combination of lack of quality control (products whose tests fail out of the box when you run them separately) or lack of cleanup in testing code. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
On 5 Oct 2005, at 14:49, Lennart Regebro wrote: On 10/5/05, Jens Vagelpohl [EMAIL PROTECTED] wrote: from the instance home will test all products in that instance home. Often doesn't work, the different products tests will walk all over each other. But setting up a script that walks all the Product directories and runs the test for the separately shouldn't be a problem. This is correct, but you're just papering over a deeper problem, namely bad cleanup in some unit tests. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
Chris Withers said: Paul Winkler wrote: Seems like the only *useful* way I can get it to run is with a loop that does bin/zopectl test once for each subdirectory of Products that I actually care about, taking care to ignore problematic third-party products. Which is OK I guess, but I'd prefer not to have to skip anything. I have a python script that builds a big command line to test.py of the form: bin/zopectl test Products/Product1|Products/Products2|etc ...it's not pretty, but it does work, and lets you exclude geb0rken products like Archetypes from test runs... Aha, that's much like what I'm doing except that yours runs in one go and thus exposes tests that don't clean up after themselves. Which is a Good Thing if I can just find and fix all the nasty things... - Some third-party products (e.g. CMF) ship with tests that just plain fail. Really? That's not been the case for a long time. When trying to test a Plohn setup, I found some of the Plohn tests were causing CMF tests to fail if the CMF tests were run in the same batch, 'cos they no doubt leave lots of crap lying around :-( If I install CMF 1.4.8 and nothing else, and run ./bin/zopectl test, I get 1 error and 1 failure. Why are you using something so ancient? ;-) Irrelevant to this discussion, but A) it's hardly ancient, and B) upgrading to 1.5.x has been postponed indefinitely as it breaks a lot of old things (some mine, some third-party) and I have not had any time to devote a day or three to resolving the incompatibilities. Anyway, as somebody replied to me in zope-cmf list last night, the breakage is fixed on the 1.4 branch and there will likely be a 1.4.9 release. - Some of my own Product tests run fine in isolation but break when run alongside other installed product tests. Then either your tests or the other tests are leaving turds... Indeed. I wish it was easier to diagnose and resolve problems like this. The problem is that tests seem to be run in alphabetical order, I don't know of a way to force another order, and so I'm left with adding and removing things from Products to see what minimal set of products will trigger the error - which is a crude technique since some products' tests will not run at all unless some other product is present (e.g. you can't run CMFDefault tests if CMFCore is not present). - Does this jibe with others' experience? Yes. Thanks. I'm always hoping that things like this are just me doing something stupid and the problem will go away with some trivial tweak I've overlooked... sadly, no such luck. - Is there a better (or best) way to run all my Product tests? If you find it, let me know! :-) Will do :) -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
Jens Vagelpohl said: On 5 Oct 2005, at 14:49, Lennart Regebro wrote: On 10/5/05, Jens Vagelpohl [EMAIL PROTECTED] wrote: from the instance home will test all products in that instance home. Often doesn't work, the different products tests will walk all over each other. But setting up a script that walks all the Product directories and runs the test for the separately shouldn't be a problem. This is correct, but you're just papering over a deeper problem, namely bad cleanup in some unit tests. Indeed. Anybody have tips on troubleshooting this? I'm currently finding it really annoying to even find which tests are the culprit(s). All I've been doing is: - try every combination of products (ugh, combinatorial explosion) - run tests with -vv so I know for sure which tests have already run when the problem occurs. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
Alan Milligan said: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Winkler wrote: The problem is that tests seem to be run in alphabetical order, I don't know of a way to force another order, and so I'm left with adding and removing things from Products to see what minimal set of products will trigger the error - which is a crude technique since some products' tests will not run at all unless some other product is present (e.g. you can't run CMFDefault tests if CMFCore is not present) def test01SomeName() ... def test02AnotherName() ... Well, at the individual test level, yeah, but I'm talking at the suite or even module level. I'm hardly going to go renaming a product folder and then have to globally change the product name in all referring code, that would be a huge mess. (Sure, I could use sed or whatever, but make one little mistake and you might have to wipe out your sandbox and start over from square one...) Would be nice if test.py allowed you to specify an order to run suites in. I'm going to go look and see how hard that would be to hack in. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Testing products: Testing all in one batch
On 10/5/05, Jens Vagelpohl [EMAIL PROTECTED] wrote: This is correct, but you're just papering over a deeper problem, namely bad cleanup in some unit tests. Sure, but that's how reality looks. It's unfortunate, but there ya go. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
