Hi Tres-- I think this is a case of us having a violent agreement :) Sorry if you get this twice -- my first attempt to send appears to have disappeared into the aether.
I agree completely that minimalist test rigs with dummy components are a good fit for some things. However, the point I was trying to make (perhaps not very clearly) is that (1) such tests come at a cost, and (2) they are not appropriate in _all_ cases; there are a lot of cases for which CMFTestCase-like tests are a better fit. See comments interspersed below. On Thu, 2005-10-06 at 11:47 -0400, Tres Seaver wrote: > > > > * As you note, dummy components take a lot of time to write. > > Not necessarily. They *do* require some knowledge of the API of the > thing they are fronting for, as well as a sense of what the calling test > needs. > > > * Dummy components create the need for new tests to ensure that the dummy > > components' functionality really does match that of the components they > > are replacing. Do we have such tests in the CMF? I'm not sure we do. > > I don't think we need to test the tests. The point of the dummies is to > emulate the published API (the interface) of the tool / content they are > replacing. Often, they won't actually *do* the work required, and may > in fact have extra stuff in them to make testing the *caller* easier. Yes, instrumenting dummy components can offer some real advantages. I am definitely not saying that one should never do these kinds of things! My concern is that the CMF's API evolves over time. If you have a bunch of dummy components, you have a bunch of things that can get out of sync. If you forget to update a dummy component's API, you could have tests passing that would fail with the real components. > > * Dummy components create the need for additional documentation. The > > absence of such documentation creates barriers to test writing and, as a > > result, to the contribution of code to the CMF. > > Nope. Dummy components do *not* need documentation. Their purpose > should be clear from use / naming, and their API is supposed to be the > same as the (already documented, we assume). The price of maintenance > (occasionally having to extend / fix the jig) is a necessary I personally find the existing dummy components to be rather obscure. Perhaps my understanding of the deep innards of the CMF is simply insufficient. I don't think there needs to be extensive documentation, but coming up with the incantation needed to, say, produce a content object with a view in a skin is not entirely straightforward. The more difficult it is to write tests, the fewer you'll get. > > At some point I think we have to trust the stack. > > I do not believe that "trusting the stack" makes senses when trying to > test a component of the stack. If you are writing tests for an > application (or higher layer) which *uses* the stack, then you can > safely trust it. For instance, I'm willing to use OFS.SimpleItem and > OFS.Folder when building out a test jig, because they belong to a lower > layer of the stack, and have their own tests. Right. I think we are in agreement here. > Such assumptions don't create unwanted dependencies, true. They may or > may not make for useful tests: > > - If the "trusted" component has no side effects which might affect > this or later tests; > > - If the "trusted" component does not make unwarrented assumptions > about the state of the system; > > - If the test being written does not need to "instrument" the > component in order to write a better / clearer / more comprehensive > test of its target component. Of course. On the other hand, dummy components make their own set of assumptions about the state of the system, which may or may not hold in a real production system. And dummy components can have bugs that can mask other problems. I'm not arguing that dummy components are bad or should not be used. I just think that CMFTestCase-type tests have an important place as well. I think it would be possible to construct something like CMFTestCase that would assume a very stripped down CMFCore site. That would make test writing for CMFDefault-level things much simpler. And I think CMFTestCase is potentially quite useful for people doing pure CMF sites. > Timing *may* be a red herring; the issue is likely worse for folks > trying to run tests on machines with less-than-blazing CPUs. Yes. My development box is no speed demon! > THere is a > classic back-and-forth in the test-driven development community > (documented by Beck and others) in which people write more and more > tests, until the run-time for the entire suite becomes so painful that > people begin avoiding running them all; the team then has to stop and > profile / refactor the tests themselves, in order to remove that burden. The desire for speed creates a number of tradeoffs. 1) One can invest time in optimizing tests. I think a bit of investment is warranted, but after a point I think developer effort is better spent writing code and tests. Where that point is is an open question. 2) One can write fewer tests. My preference is for more tests at the expense of speed, especially since the cost of CPU has been dropping rapidly over time. I'm sure you would agree. 3) One can run tests less frequently. I agree with you that this is an undesirable outcome. Three cheers for whoever added the "zopectl test" feature with the additional options for selecting a particular directory and a particular test. In my own work I tend to run a focused set of tests frequently while working on a single feature, then I periodically run all tests to make sure I haven't created any side effects. That may be anathema to the XP crowd, but I find it works well for me. Maybe one solution would be create some kind of sensible partitioning of tests? Tests are already partitioned by module; why not partition further so people can focus their test cycles on the things that matter most to the work at hand? > Here are timings for the stock CMF components and Plone on my box: > > Product # tests Wall-clock (s) > --------- ------- -------------- > CMFCore 382 28.775 > CMFDefault 164 2.980 > CMFActionIcons 11 0.002 > CMFCalendar 23 1.636 > CMFTopic 58 1.898 > CMFSetup 341 2.028 > DCWorkflow 10 0.025 I don't see the Plone numbers, but I'm guessing that you probably got numbers along the lines of CMFPlone ~1500 ~175 sec Plone's tests are slow, but there are a _lot_ of them. I think that's an important thing to note. There are a lot of tests for Plone in part because they are easy to write. I'd rather have full(er) coverage with slow tests than less full coverage with fast ones. Note that I'm _not_ claiming that CMF's tests are insufficient or that Plone's are complete. Rather, I'm saying that good tools can help everyone get more tests written and more good code into the CMF. > My guess is that CMFCore's tests are ripe for such a refactoring (there > is a noticeable lag of a couple of seconds several times during the test > run, for instance). > > False dependencies, poor separation of concerns, and poor test coverage > are real issues with using a "functional" test jig where a unit test jig > would be more appropriate. I agree completely. There are definitely places where narrow, focused tests are the right way to go. More "functional" style tests do have their disadvantages. On the flip side, though, minimalist test tools that require developers to jump through extra hoops can raise barriers to entry for community participation and can hinder development. Open source is as much social engineering as software engineering; simplifying tests are one way to address some needs in the latter area. Geoff _______________________________________________ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See http://collector.zope.org/CMF for bug reports and feature requests