Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
Exactly! ZopeTestCase (CMFTestCase, PloneTestCase) was designed for testing *applications*. As you say, when I write a CMF application I should be able to assume the CMF below works. In this case a "fat fixture" makes perfect sense (to me). Also, I firmly believe application developers should not need to dig deep into the framework just to be able to write some tests; this was my motivation for starting ZTC in the first place. I am not in the business of shoving CMFTestCase down anybody's throat. CMFCore.tests.base has everything CMF-the-framework needs, and SecurityRequestTest provides basically the same fixture you would get from using ZTC (app object, transaction, request, and security context). Whether tests you write turn out to be unit or integration tests does not depend on the test framework, IMO. It depends on where you draw the line in the stack. Stefan On 6. Okt 2005, at 17:47, Tres Seaver wrote: I do not believe that "trusting the stack" makes senses when trying to test a component of the stack. If you are writing tests for an application (or higher layer) which *uses* the stack, then you can safely trust it. For instance, I'm willing to use OFS.SimpleItem and OFS.Folder when building out a test jig, because they belong to a lower layer of the stack, and have their own tests. -- Anything that happens, happens. --Douglas Adams ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
Hi Tres-- I think this is a case of us having a violent agreement :) Sorry if you get this twice -- my first attempt to send appears to have disappeared into the aether. I agree completely that minimalist test rigs with dummy components are a good fit for some things. However, the point I was trying to make (perhaps not very clearly) is that (1) such tests come at a cost, and (2) they are not appropriate in _all_ cases; there are a lot of cases for which CMFTestCase-like tests are a better fit. See comments interspersed below. On Thu, 2005-10-06 at 11:47 -0400, Tres Seaver wrote: > > > > * As you note, dummy components take a lot of time to write. > > Not necessarily. They *do* require some knowledge of the API of the > thing they are fronting for, as well as a sense of what the calling test > needs. > > > * Dummy components create the need for new tests to ensure that the dummy > > components' functionality really does match that of the components they > > are replacing. Do we have such tests in the CMF? I'm not sure we do. > > I don't think we need to test the tests. The point of the dummies is to > emulate the published API (the interface) of the tool / content they are > replacing. Often, they won't actually *do* the work required, and may > in fact have extra stuff in them to make testing the *caller* easier. Yes, instrumenting dummy components can offer some real advantages. I am definitely not saying that one should never do these kinds of things! My concern is that the CMF's API evolves over time. If you have a bunch of dummy components, you have a bunch of things that can get out of sync. If you forget to update a dummy component's API, you could have tests passing that would fail with the real components. > > * Dummy components create the need for additional documentation. The > > absence of such documentation creates barriers to test writing and, as a > > result, to the contribution of code to the CMF. > > Nope. Dummy components do *not* need documentation. Their purpose > should be clear from use / naming, and their API is supposed to be the > same as the (already documented, we assume). The price of maintenance > (occasionally having to extend / fix the jig) is a necessary I personally find the existing dummy components to be rather obscure. Perhaps my understanding of the deep innards of the CMF is simply insufficient. I don't think there needs to be extensive documentation, but coming up with the incantation needed to, say, produce a content object with a view in a skin is not entirely straightforward. The more difficult it is to write tests, the fewer you'll get. > > At some point I think we have to trust the stack. > > I do not believe that "trusting the stack" makes senses when trying to > test a component of the stack. If you are writing tests for an > application (or higher layer) which *uses* the stack, then you can > safely trust it. For instance, I'm willing to use OFS.SimpleItem and > OFS.Folder when building out a test jig, because they belong to a lower > layer of the stack, and have their own tests. Right. I think we are in agreement here. > Such assumptions don't create unwanted dependencies, true. They may or > may not make for useful tests: > > - If the "trusted" component has no side effects which might affect > this or later tests; > > - If the "trusted" component does not make unwarrented assumptions > about the state of the system; > > - If the test being written does not need to "instrument" the > component in order to write a better / clearer / more comprehensive > test of its target component. Of course. On the other hand, dummy components make their own set of assumptions about the state of the system, which may or may not hold in a real production system. And dummy components can have bugs that can mask other problems. I'm not arguing that dummy components are bad or should not be used. I just think that CMFTestCase-type tests have an important place as well. I think it would be possible to construct something like CMFTestCase that would assume a very stripped down CMFCore site. That would make test writing for CMFDefault-level things much simpler. And I think CMFTestCase is potentially quite useful for people doing pure CMF sites. > Timing *may* be a red herring; the issue is likely worse for folks > trying to run tests on machines with less-than-blazing CPUs. Yes. My development box is no speed demon! > THere is a > classic back-and-forth in the test-driven development community > (documented by Beck and others) in which people write more and more > tests, until the run-time for the entire suite becomes so painful that > people begin avoiding running them all; the team then has to stop and > profile / refactor the tests themselves, in order to remove that burden. The desire for speed creates a number of tradeoffs. 1) One can invest time in optimizing tests. I think a bit of investme
[Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Geoff Davis wrote: > On Wed, 05 Oct 2005 12:34:25 +0200, Lennart Regebro wrote: > > >>Any test including PortalTestCase should really not be seen as a unit >>test, but a fucntional test. ;) If we could put in some effort of >>making a minimal dymmy-portal that can be deleted and recreated very >>quickly, then that would be very interesting. I would assume that that >>involved a lot of work though... > > > CMFTestCase creates a minimal portal that can be deleted and recreated > relatively quickly. But it actually is even smarter than that: it uses > the transaction machinery in such a way that it only has to create and > delete the test site once. See my reply to Chris for an explanation. > > The nice thing about CMFTestCase is that it creates an actual CMF site, > not some dummy site whose functionality may or may not be equivalent to > that of a real CMF site. I think there is a place for using really > stripped-down dummy components. However, widespread use of dummy > components comes with some real headaches: > > * As you note, dummy components take a lot of time to write. Not necessarily. They *do* require some knowledge of the API of the thing they are fronting for, as well as a sense of what the calling test needs. > * Dummy components create the need for new tests to ensure that the dummy > components' functionality really does match that of the components they > are replacing. Do we have such tests in the CMF? I'm not sure we do. I don't think we need to test the tests. The point of the dummies is to emulate the published API (the interface) of the tool / content they are replacing. Often, they won't actually *do* the work required, and may in fact have extra stuff in them to make testing the *caller* easier. > * Dummy components create the need for additional documentation. The > absence of such documentation creates barriers to test writing and, as a > result, to the contribution of code to the CMF. Nope. Dummy components do *not* need documentation. Their purpose should be clear from use / naming, and their API is supposed to be the same as the (already documented, we assume). The price of maintenance (occasionally having to extend / fix the jig) is a necessary > At some point I think we have to trust the stack. I do not believe that "trusting the stack" makes senses when trying to test a component of the stack. If you are writing tests for an application (or higher layer) which *uses* the stack, then you can safely trust it. For instance, I'm willing to use OFS.SimpleItem and OFS.Folder when building out a test jig, because they belong to a lower layer of the stack, and have their own tests. > After all, we don't go > around writing dummy versions of python modules such as httplib. CMFCore > should be able to assume Zope; CMFDefault modules should be able to assume > CMFCore components; products built on CMFDefault should be able to assume > it, etc. Such assumptions don't create unwanted dependencies, true. They may or may not make for useful tests: - If the "trusted" component has no side effects which might affect this or later tests; - If the "trusted" component does not make unwarrented assumptions about the state of the system; - If the test being written does not need to "instrument" the component in order to write a better / clearer / more comprehensive test of its target component. > I think the speed issue is a red herring. I just timed Plone's tests > (almost all of which use PloneTestCase) and CMFCore's tests (all of which > use stripped down dummy components). The results: > > Plone tests: 0.14 sec/test > CMFCore tests: 0.09 sec/test > > The dummy components really aren't saving much time. If you spent the > same amount of work on customer projects that you would spent writing, > documenting, and maintaining a set of good dummy components, I am sure you > could buy a very, very fast computer that would run the tests in no time. Timing *may* be a red herring; the issue is likely worse for folks trying to run tests on machines with less-than-blazing CPUs. THere is a classic back-and-forth in the test-driven development community (documented by Beck and others) in which people write more and more tests, until the run-time for the entire suite becomes so painful that people begin avoiding running them all; the team then has to stop and profile / refactor the tests themselves, in order to remove that burden. Here are timings for the stock CMF components and Plone on my box: Product # tests Wall-clock (s) - --- -- CMFCore 382 28.775 CMFDefault 1642.980 CMFActionIcons 110.002 CMFCalendar 231.636 CMFTopic 581.898 CMFSetup3412.028 DCWorkflow 100.025 My guess is that
Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
OK, so I misunderstood the actual problem Paul was reporting... Can someone explain it for a stupid person like me? ;-) Chris Stefan H. Holek wrote: Oh, setupCMFSite is sufficiently clever, it's the CMF tests that aren't ;-) Stefan On 5. Okt 2005, at 10:45, Chris Withers wrote: Stefan H. Holek wrote: I see two options: a) Use a different portal name portal_name = 'mysite' CMFTestCase.setupCMFSite(portal_name) Can setupCMFSite not do something more clever if it finds the object already there? -- Anything that happens, happens. --Douglas Adams -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
Lennart Regebro said: > On 10/5/05, Chris Withers <[EMAIL PROTECTED]> wrote: >> Stefan H. Holek wrote: >> > I see two options: >> > >> > a) Use a different portal name >> > >> > portal_name = 'mysite' >> > CMFTestCase.setupCMFSite(portal_name) >> >> Can setupCMFSite not do something more clever if it finds the object >> already there? It simply doesn't do anything in that case. >> > b) Use getPortal() to create a new portal per test >> > >> > def getPortal(self): >> > manage_addCMFSite(self.app, portal_name) >> > return getattr(self.app, portal_name) >> >> hahaha, and watch your tests take hours to run ;-) Maybe that way, but setupCMFSite() is pretty quick. > Any test including PortalTestCase should really not be seen as a unit > test, but a fucntional test. ;) Regardless of which, we still need to run it :) > If we could put in some effort of > making a minimal dymmy-portal that can be deleted and recreated very > quickly, then that would be very interesting. I would assume that that > involved a lot of work though... CMFTestCase's setupCMFSite() is already pretty quick. -- Paul Winkler http://www.slinkp.com ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
Oh, setupCMFSite is sufficiently clever, it's the CMF tests that aren't ;-) Stefan On 5. Okt 2005, at 10:45, Chris Withers wrote: Stefan H. Holek wrote: I see two options: a) Use a different portal name portal_name = 'mysite' CMFTestCase.setupCMFSite(portal_name) Can setupCMFSite not do something more clever if it finds the object already there? -- Anything that happens, happens. --Douglas Adams ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
On 10/5/05, Chris Withers <[EMAIL PROTECTED]> wrote: > Stefan H. Holek wrote: > > I see two options: > > > > a) Use a different portal name > > > > portal_name = 'mysite' > > CMFTestCase.setupCMFSite(portal_name) > > Can setupCMFSite not do something more clever if it finds the object > already there? > > > b) Use getPortal() to create a new portal per test > > > > def getPortal(self): > > manage_addCMFSite(self.app, portal_name) > > return getattr(self.app, portal_name) > > hahaha, and watch your tests take hours to run ;-) > > Chris Any test including PortalTestCase should really not be seen as a unit test, but a fucntional test. ;) If we could put in some effort of making a minimal dymmy-portal that can be deleted and recreated very quickly, then that would be very interesting. I would assume that that involved a lot of work though... -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
Re: [Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
Stefan H. Holek wrote: I see two options: a) Use a different portal name portal_name = 'mysite' CMFTestCase.setupCMFSite(portal_name) Can setupCMFSite not do something more clever if it finds the object already there? b) Use getPortal() to create a new portal per test def getPortal(self): manage_addCMFSite(self.app, portal_name) return getattr(self.app, portal_name) hahaha, and watch your tests take hours to run ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
[Zope-CMF] Re: CMFTestCase: Best way to create the CMF site?
I see two options: a) Use a different portal name portal_name = 'mysite' CMFTestCase.setupCMFSite(portal_name) b) Use getPortal() to create a new portal per test def getPortal(self): manage_addCMFSite(self.app, portal_name) return getattr(self.app, portal_name) Stefan On 5. Okt 2005, at 01:08, Paul Winkler wrote: At one time, I naively had it set up in getPortal() but that seems to have worked only in one product and only by bizarre coincidence as it failed in every other product test I tried it with. (Random assortment of BadRequest and ReadConflictErrors, I was never able to sort it all out.) So that seems unlikely to be the Right Thing (tm). -- Anything that happens, happens. --Douglas Adams ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests
