On 15/02/2010 17:05, Michael Foord wrote:
[snip]
This is also an interesting point. The 'naive' implementation, which I
think I prefer, only runs the setUpModule of modules actually
containing tests. Similarly setUpClass is only called on classes with
actual tests, although they may call up to base class implementations.
This has a couple of consequences, particularly for setUpModule. It
makes the basic rule:
* only use setUpModule for modules actually containing tests
* don't mix concrete TestCases (with tests) in the same module (using
setUpModule) as base classes for other tests
The use case (that I can think of) that isn't supported (and of course
all this needs to be documented):
* Having setUpModule / tearDownModule in modules that define base
classes and expecting them to be called around all tests that inherit
from those base classes
Having this in place makes the implementation simpler. If we
explicitly document that this rule may change and so users shouldn't
rely on setUpModule not being called for modules containing base
classes, then we are free to rethink it later. Not having this
restriction at all is possible, it just requires more introspection at
TestSuite creation / ordering time.
Note that setUpClass on a base class maybe called several times if
several base classes inherit and all call up to the base class
implementation. As it will be a class method the cls argument will be
different for each call.
Another question. If we are implementing TestCase.setUpClass as an
additional test then should it be reported *even* if it is only the
default (empty) implementation that is used?
The reason to have setUpClass implemented as a test is so that you can
report the failure *before* you run all the tests. Lots of unit test
users want a consistent number of tests every run - so we shouldn't
insert an extra test only on fail. The other alternative is to report
a setUpClass failure as part of the first test that depends on it -
this makes the implementation more, complex (having a pseudo-test
represent setUpClass / tearDownClass is convenient for ordering and
isolating the 'magic' in one place - the rest of the test running
infrastructure doesn't need to know about setUpClass or Module).
If we do add a default setUpClass test for all TestCases it means
extra noise for test runs that don't use the new feature. Saying no it
shouldn't be shown means that we have to introspect test classes to
see if they inherit setUpClass from TestCase or from some intermediate
base class. Not hard just an extra complexity. One solution would be
for TestCase *not* to have default implementations, but it is probably
nicer for them to appear in the API.
I guess my preferred approach is to have a default implementation, but
not to create pseudo-tests for them if they aren't used.
One place to implement this is in the TestLoader (specifically in
loadTestsFromModule and loadTestsFromTestCase) - especially as this is
the place where test ordering is currently provided by unittest. The
TestSuite may not need to change much. This isn't compatible with test
frameworks that build custom suites without going through
loadTestsFromModule - e.g. modules implementing load_tests that replace
the standard test suite with a new suite with specific TestCases and
still expect setUpModule / tearDownModule to be used [1]. Perhaps an API
hook to make this easy?
TestCase would need a change in run(...) to support automatic
SetupFailed test failing when setUpClass / module fails.
This *isn't* compatible with custom TestSuites that reorder tests. The
alternative is for TestSuite.run(...) to change to support setUpClass /
setUpModule by adding them at test run time by introspecting contained
tests. That would make setUpClass (etc) incompatible with custom
TestSuite implementations that override run. It would be compatible with
TestSuites that do reordering but that don't overload run(...) -
although the resulting test order may not be 'optimal' (the setUp and
tearDown may end up separated farther in time than desired but at least
they would work). Perhaps this approach is less likely to break?
(Manually reordering tests or adding new tests to a TestSuite wouldn't
break setUpClass as the TestSuite only adds them at run(...) time rather
than placing them in the suite where the 'user' can move them around).
All the best,
Michael
[1] or even load_tests that just addTests to the standard suite but
still need tearDownModule to be run *after* all the tests. Even with the
implementation in loadTestsFrom* it seems like the TestSuite may need to
know something about the order. Perhaps we could have
TestLoader.orderTests - the loader is available to load_tests functions.
All the best,
Michael
testresources very neatly sidesteps this problem by just providing an
API to say "this test case depends on that test resource", without
relying on the grouping of tests within classes, modules, or
packages. Of course you can just define a class-level or
module-level resource and then have all your tests depend on it,
which gives you the behavior of setUpClass and setUpModule in a more
general way.
-glyph
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of
your employer, to release me from all obligations and waivers arising from any
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap,
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in perpetuity, without
prejudice to my ongoing rights and privileges. You further represent that you
have the authority to release me from any BOGUS AGREEMENTS on behalf of your
employer.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
