Hello all,
unittest has seen quite a few new features since Python 2.6. For those
of you who might have missed the announcements in other places,
unittest2 is a backport of the new features in unittest to work with
Python 2.4-2.6. It is already being used for the development of distutils2:
http://pypi.python.org/pypi/unittest2/
<http://pypi.python.org/pypi/unittest2/>
In other news, class and module fixtures (setUpClass / tearDownClass /
setUpModule / tearDownModule) are now implemented in unittest (in trunk
- not yet merged to py3k). These features are tested but I'm sure there
are some lurking bugs or oddities, so please try them out. I have not
yet added documentation for them; I'll pull it out from this email as a
starting point.
I'd rather this thread didn't become *another* debate on the merit of
these features, but perhaps that is too much to hope for.
Below are some notes on how class and module fixtures work.
Class and module level fixtures are implemented in unittest.TestSuite.
When the test suite encounters a test from a new class then
tearDownClass from the previous class (if there is one) is called,
followed by setUpClass from the new class.
Similarly if a test is from a different module from the previous test
then tearDownModule from the previous module is run, followed by
setUpModule from the new module.
After all the tests have run the final tearDownClass and tearDownModule
are run.
Note that shared fixtures do not play well with [potential] features
like test parallelization and they break test isolation. They should be
used with care.
setUpClass and tearDownClass
--------------------------------------
These must be implemented as class methods.
import unittest
class Test(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls._connection = createExpensiveConnectionObject()
@classmethod
def tearDownClass(cls):
cls._connection.destroy()
If you want the setUpClass and tearDownClass on base classes called then
you must call up to them yourself. The implementations in
unittest2.TestCase are empty.
If an exception is raised during a setUpClass then the tests in the
class are not run and the tearDownClass is not run. Skipped classes will
not have setUpClass or tearDownClass run.
A setUpClass that raises a unittest2.SkipTest exception will currently
be reported as an error rather than a skip (although the effect is the
same). This will be fixed at some point in the future.
setUpModule and tearDownModule
--------------------------------------------
These should be implemented as functions.
def setUpModule():
createConnection()
def tearDownModule():
closeConnection()
If an exception is raised in a setUpModule then none of the tests in the
module will be run and the tearDownModule will not be run.
A setUpModule that raises a unittest.SkipTest exception will currently
be reported as an error rather than a skip (although the effect is the
same). This will be fixed at some point in the future.
The Gory Details
----------------------
The default ordering of tests created by the unittest test loaders is to
group all tests from the same modules and classes together. This will
lead to setUpClass / setUpModule (etc) being called exactly once per
class and module. If you randomize the order, so that tests from
different modules and classes are adjacent to each other, then these
shared fixture functions may be called multiple times in a single test run.
This particular issue was a point of some disussion. Shared fixtures are
not intended to work with suites with non-standard (incompatible)
ordering (a BaseTestSuite still exists for frameworks that don't want to
support shared fixtures), so this is already a pathological situation.
There are a couple of things unittest could do in this situation:
1) Work out the best order to run the setUp and tearDowns, based on
the position of the first and last test from each class / module in the
suite.
2) Run setUp and tearDowns appropriately if consecutive tests are
from different classes and modules.
If your setUps do unrelated things then strategy 1 will be the best one,
but (so long as your setups are repeatable) 2 will be fine just slower.
If your setups do related things, for example pushing large test
datasets into the same database, then having multiple setups active
before the corresponding teardowns have been run will actually do the
wrong thing and your tests will be broken. In addition strategy 1 relies
on unrolling the test suite and introspecting all tests prior to the
run, not all custom test suites implementations support this.
Given that strategy 2 was substantially easier to implement, and less
likely to cause broken test runs, that was what I went with. It is of
course possible to add alternative strategies, plus the metadata to
support them, if there proves to be a substantial need for this. One of
the major use cases is for test order randomization. It would be easy to
add a 'randomize' method to the suite that randomizes the order of test
modules in a suite, the order of test classes within modules and the
order of tests within a class. This would allow for tests to be run in a
random order whilst remaining compatible with shared fixtures.
If there are any exceptions raised during one of the shared fixture
functions the test is reported as an error. Because there is no
corresponding test instance an _ErrorHolder object (that has the same
interface as a TestCase) is created to represent the error. If you are
just using the standard unittest test runner then this detail doesn't
matter, but if you are a framework author it may be relevant.
I have a few minor features I'd like to add to unittest before 2.7 beta
- standard out capturing, fail fast and ctrl-c handling (first ctrl-c
will wait for current test to exit, second will force an exit - results
will still be reported). After that the *really* big problem for
unittest is the extensibility story.
I'll be working on extensibility points for unittest that allow people
to implement extensions without having to override functionality. This
will take a fair bit of discussion and experimentation, so I will
probably trial this in unittest2 first. It would be nice to have
something in place for the release of 3.2, but that may be ambitious.
All the best,
Michael Foord
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your
employer, to release me from all obligations and waivers arising from any and all
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS
AGREEMENTS") that I have entered into with your employer, its partners, licensors,
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me from any BOGUS AGREEMENTS
on behalf of your employer.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
