Hello, part two. :)
* Uli Fouquet <u...@gnufix.de> [2011-03-24 01:05]: > A big advantage of test layers over `pytest` testing scopes might be > that you can spread your tests associated to a certain layer over many > files/modules/packages as you like and the setup/teardown will > nevertheless only be performed once for each layer > > Compared to Zope test layers I came to the conclusion that there is not > much like this concept already in `pytest` and the behaviour of test > layers can not easily be faked. I fully agree, the test layer concept of zope.testrunner provides some very nice properties, most notably: 1) Shared fixture setup. It is shared across many TestCases and created only once per test run. (And it is independent of how the tests are organized into modules or whatever.) 2) Stacking of fixture setups. My canonical example: First, we initialize the application, load ZCML, prepare databases, etc. and talk to it with zope.testbrowser. That's the first layer. Then we add a HTTP port with gocept.selenium on another layer and do the rest with a real browser -- but the whole, possibly expensive, application setup *does not have to be done again*. 3) Orthogonal composition, i.e. you can have small, well defined fixture units that you can then combine in a separate step to form the actual setup for your tests. This enables for example that library packages like zope.component could provide test infrastructure for themselves, and application packages could pull all the test infrastructure together that they need -- not like zope.app.testing for example, which is a giant monolith that sets up everything at once. I think that these properties are very valuable, and I haven't seen them anywhere else so far. I haven't looked at py.test closely, yet, but from what I've seen, and what you wrote, it doesn't seem to offer these things right now. > Would it make sense to bring the Zope layer concept into `pytest`? > > Are there already possibilities to mimic testlayer-like behaviour in > `pytest` which we simply overlooked? An honest counter question: Why not use zope.testrunner? What advantages does py.test offer? But your email brings into focus a question that I've been thinking about lately: What's the future direction for test fixture setup? Because, while test layers are nice because they have the above properties, I'm not too happy with the current implementation, namely the use (or is it abuse? ;-) of __bases__ and __name__, which leads very naturally to not-so-helpful implementation patterns. As a concrete example, some of the test layers of ZTK packages, most notably the successors of zope.app.testing.functional in zope.component, zope.app.appsetup, zope.app.wsgi, are implemented right now in a way that does not have properties 2+3. The most straightforward way to improve that situation would be to use plone.testing, since that provides a layer implementation that is both sane and easy to use, and has all the nice properties. (Maybe even fold some of it into zope.testrunner itself.) But then I realized, that's a major undertaking, so we better think about what the actual goals are before changing lots of stuff. And then I heard people expressing interest in other test runners. (py.test and nose seem to be the biggest players, stdlib's unittest/unittest2 seems to be woefully behind in terms of test selection and other features like coverage or debugging. I for one happen to like zope.testrunner, but if there's a good alternative, why not combine our efforts?) I also found that it is rather hard to explain how to use test layers in a way so you actually get the above properties, so I started wondering whether there was a cleaner way to get them. So I guess I have three questions: 1. (The starting point:) How can we improve the test infrastructure situation of the ZTK packages? 2. (The main thing:) Are test layers, both the concept and the implementation as epitomized by plone.testing, the way to go or is there a cleaner alternative? 3. (But only tangentially, I don't want to sidetrack the discussion at hand nor start a flamewar:) Is there a compelling alternative to zope.testrunner that would make it worthwile to think about either generalizing the fixture support or even migrating away from zope.testrunner? Wolfgang -- Wolfgang Schnerring · w...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )