Ok,
So I'm late with this commentary (to the point that the test runner, etc are already checked in). So far I like what I see.
My only other comment is that I agree that we should be adding test methods in preference to test modules.
Ted
On Jan 26, 2005, at 1:03 PM, Phillip J. Eby wrote:
Hi folks. Lisa has asked me to put together a unit test runner that will allow selective running of multiple unit tests, for example to run just fast unit tests, or to run all tests including functional tests. I have a couple of different ways to do that, but I'd like to get some input from the team as to what they'd like to see in such a runner. But before I get to the questions, I need to lay out some background on Chandler's current test setup, and the options that are available for how to proceed.
Currently, the way most of Chandler's tests are structured, you can run individual test modules with either hardhat or unittest.py, e.g.:
unittest.py repository.tests.TestText
will run that specific test module, or:
unittest.py repository.tests.TestText.TestText.testBZ2Compressed
to run an individual test within the module. You can also use hardhat to collect all test modules named 'Test*.py'.
So, I gather that what Lisa's asking for is the ability to also run a selected subset of the total collection of tests, such as just the "strict" unit tests that test subsystems in isolation, with individual runtimes of about 10ms or less. (Most of Chandler's current tests are technically speaking functional or integration tests, as they do not test subsystems in isolation from each other. These types of tests are important also, but because they take longer to run, it's useful to also have more "isolationist" tests for use during development.)
Python's unittest module has a standard way of representing a collection of tests, called a "suite". The Python doctest module can also create suites from embedded and external doctests. (Embedded = doctests in docstrings, external = doctests in text files.) If you write a function that returns a test suite, then that function can be run with unittest.py. For example, here's an excerpt from a unit test module in PyProtocols (protocols.tests.test_dispatch):
from unittest import TestCase, makeSuite, TestSuite import doctest
# actual test classes omitted for brevity
TestClasses = (
TestGraph, TestTests, ExpressionTests, SimpleGenerics, GenericTests,
)
def test_combiners():
return doctest.DocFileSuite(
'combiners.txt', optionflags=doctest.ELLIPSIS, package='dispatch',
)
def test_suite(): return TestSuite( [test_combiners(), ] + [makeSuite(t,'test') for t in TestClasses] )
The two functions here, 'test_suite' and 'test_combiners' are functions that return unittest "test suites". One of them creates an external doctest that runs the tests in 'combiners.txt' (a tutorial text file), and the other creates a test suite that combines that test with test suites for each of the other test classes in the module.
Thus, to run all 50 unit tests specified by this test module, I can now use:
unittest.py protocols.tests.test_dispatch.test_suite
Or to run just the doctest, I can use:
unittest.py protocols.tests.test_dispatch.test_combiners
And of course I can still also specify an individual test case class or method.
There are also a couple of other ways to gather tests. For example, this code defines a function that returns a suite of all the test classes found in every module of the 'repository.tests' package:
from unittest import defaultTestLoader, TestSuite import repository.tests
testNames = """ TestAlias TestBZ2 TestBinary TestDeepCopyRef TestDelete TestIndexes TestItems TestKinds TestLiteralAttributes TestMerge TestMixins TestMove TestPerfWithRSS TestPersistentCollections TestRedirectToOrdering TestRefDictAlias TestReferenceAttributes TestRepository TestRepositoryBasic TestSkipList TestText TestTypes """.split()
def test_suite():
return TestSuite(
[defaultTestLoader.loadTestsFromName(name, repository.tests)
for name in testNames]
)
Of course, the names in 'testNames' could name individual test classes, methods, or suite-generating functions. They can also be absolute module names instead of ones relative to a particular location.
So as you can see, there are a lot of potential ways we could organize this, with different tradeoffs depending on what tests people want to run and how often, not to mention how often they need to change the list of the tests. There are even tradeoffs depending on how many tests you put in a single module.
Currently, I notice that most Chandler tests have only one test class per module, and many of those actually only have one test method per test case. I'm guessing that this is a byproduct of how long tests take to run, coupled with the fact that hardhat (unlike unittest.py) doesn't allow selecting an individual test within a module to run.
By contrast, using unittest.py to select and run individual tests, you can actually include more tests in the same module, with no overhead penalty for doing so. This will be important as we add fast-running unit tests, as each unit test tests only a very small piece of functionality, and there are therefore usually a lot of them. For example, PyProtocols has only seven test modules, but these contain hundreds of test methods; a typical run of those tests involves maybe 320 individual test methods, each with an average of maybe half a dozen assertions being tested.
So, if we start adding unit tests (in the strict isolationist sense) to increase the coverage-to-runtime ratio of Chandler's tests, it's likely that we'll tend towards putting more tests per class and more classes per module than is currently done. So, if we set up a test-gathering strategy that works well for the current situation, it won't necessarily continue to work long-term.
On the other hand, I don't want to burden anybody with a test-gathering style that would work well in the future, but which might seem tedious for today's more limited needs. So, it seems like the best thing for me to do is throw all this out here for discussion and find out what approach the team would prefer to take.
(P.S. I forgot to mention... right now 'application.tests' isn't a package, so it needs an '__init__.py' if you want to use 'unittest.py' to run any of the tests in it. I think all the other test locations are already packages.)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
---- Ted Leung Open Source Applications Foundation (OSAF) PGP Fingerprint: 1003 7870 251F FA71 A59A CEE3 BEBA 2B87 F5FC 4B42
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
