On Tue, 2010-02-16 at 02:09 -0500, Glyph Lefkowitz wrote:
> > > > In a recent message he was talking about either breaking > > > compatibility with TestSuite implementations that override run(), > > > or test-reordering - both of which I consider important, core > > > features of the unittest module. > > > > Well, by "breaking compatibility with custom TestSuite > > implementations that override run" I mean that is one possible place > > to put the functionality. Code that does override it will *not* stop > > working, it just won't support the new features. > > > > > Ah, I see. This doesn't sound *too* bad, but I'd personally prefer it > if the distinction were a bit more clearly drawn. I'd like frameworks > to be able to implement extension functionality without having to > first stub out functionality. In other words, if I want a test suite > without setUpClass, I'd prefer to avoid having an abstraction > inversion. +1 > > If we chose this implementation strategy there would be no > > compatibility issues for existing tests / frameworks that don't use > > the new features. > > That's very good to hear. It does however get tougher to be 'stdlib compatible' for frameworks that extend the stdlib - at least with how extensions work today. > > If tests do want to use the new features then the framework authors > > will need to ensure they are compatible with them. This seems like a > > reasonable trade-off to me. We can ensure that it is easy to write > > custom TestSuite objects that work with earlier versions of unittest > > but are also compatible with setUpClass in 2.7 (and document the > > recipe - although I expect it will just mean that TestSuite.run > > should call a single method if it exists). > > > > > This is something that I hope Jonathan Lange or Robert Collins will > chime in to comment on: expanding the protocol between suite and test > is an area which is fraught with peril, but it seems like it's > something that test framework authors always want to do. (Personally, > *I* really want to do it because I want to be able to run things > asynchronously, so the semantics of 'run()' need to change pretty > dramatically to support that...) It might be good to eventually > develop a general mechanism for this, rather than building up an > ad-hoc list of test-feature compatibility recipes which involve a list > of if hasattr(...): foo(); checks in every suite implementation. Please have a look at the testtools.TestCase.run - its incomplete, but its working towards making it possible for trial to not need to replace run, but instead provide a couple of hooks (registered during setUp) to handle what you need. What it currently offers is catching additional exceptions for you, which is a common form of extension. bzrlib is using this quite successfully, and we deleted a lot of code that overlapped the stdlib unittest run(). > > Perhaps a better idea might be to also add startTest and stopTest > > methods to TestSuite so that frameworks can build in features like > > timing tests (etc) without having to override run itself. This is > > already possible in the TestResult of course, which is a more common > > extensibility point in *my* experience. > > > > I think timing and monitoring tests can mostly be done in the > TestResult class; those were bad examples. There's stuff like > synthesizing arguments for test methods, or deciding to repeat a > potentially flaky test method before reporting a failure, which are > not possible to do from the result. I'm not sure that startTest and > stopTest hooks help with those features, the ones which really need > suites; it would seem it mostly gives you a hook to do stuff that > could already be done in TestResult anyway. Also its not really possible to 'run one thing' around a test at the moment - theres no good place (without changing tests or doing somewhat convoluted stuff) to have custom code sit in the stack above the test code - this makes it harder to handle: - profiling - drop-into-a-debugger - $other use case This is also in my hit-list of things to solve-and-propose-for-stdlib-unittest that I blogged about a while back. -Rob
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
