On 15/02/2010 20:27, Glyph Lefkowitz wrote:
On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
<gl...@twistedmatrix.com <mailto:gl...@twistedmatrix.com>> wrote:
On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
For what it's worth, I am a big fan of abusing test frameworks in
generally, and pyunit specifically, to perform every possible kind
of testing. In fact, I find setUpClass more hostile to *other*
kinds of testing, because this convenience for simple integration
tests makes more involved, performance-intensive integration tests
harder to write and manage.
That sounds odd, as if the presence of this convenience would prohibit
you from also implement other features.
Well, that is the main point I'm trying to make. There are ways to
implement setUpClass that *do* make the implementation of other
features effectively impossible, by breaking the integration
mechanisms between tests and framework, and between multiple testing
frameworks.
And I am pretty sure this is not just my over-reaction; Michael still
appears to be wrestling with the problems I'm describing.
And I appreciate your input.
In a recent message he was talking about either breaking
compatibility with TestSuite implementations that override run(), or
test-reordering - both of which I consider important, core features of
the unittest module.
Well, by "breaking compatibility with custom TestSuite implementations
that override run" I mean that is one possible place to put the
functionality. Code that does override it will *not* stop working, it
just won't support the new features.
If we chose this implementation strategy there would be no compatibility
issues for existing tests / frameworks that don't use the new features.
If tests do want to use the new features then the framework authors will
need to ensure they are compatible with them. This seems like a
reasonable trade-off to me. We can ensure that it is easy to write
custom TestSuite objects that work with earlier versions of unittest but
are also compatible with setUpClass in 2.7 (and document the recipe -
although I expect it will just mean that TestSuite.run should call a
single method if it exists).
Perhaps a better idea might be to also add startTest and stopTest
methods to TestSuite so that frameworks can build in features like
timing tests (etc) without having to override run itself. This is
already possible in the TestResult of course, which is a more common
extensibility point in *my* experience.
All the best,
Michael
I tried to write about this problem a while ago
<http://glyf.livejournal.com/72505.html> - the current extensibility
API (which is mostly just composing "run()") is sub-optimal in many
ways, but it's important not to break it.
I expect that *eventually* something will come along that is so much
better than unittest that, once matured, we'll want it in the stdlib.
I'm not sure what point you're trying to make here. I was saying
"it's not perfect, but we should be careful not to break it, because
it's all we've got". Are you saying that we shouldn't worry about
unittest's composition API, because it's just a stopgap until
something better comes along?
(Or, alternatively, eventually stdlib inclusion won't be such a big
deal any more since distros mix and match. But then inclusion in a
distro would become every package developer's goal -- and then the
circle would be round, since distros hardly move faster than Python
releases...)
But in the mean time I believe evolving unittest is the right thing to
do. Adding new methods is relatively easy. Adding whole new paradigms
(like testresources) is a lot harder, eventually in the light of the
latter's relative immaturity.
I disagree with your classification of the solutions.
First and foremost: setUpClass is not a "new method", it's a pile of
new code to call that method, to deal with ordering that method, etc.
Code which has not yet been written or tested or tried in the real
world. It is beyond simply immature, it's hypothetical. We do have an
implementation of this code in Twisted, but as I have said, it's an
albatross we are struggling to divest ourselves of, not something we'd
like to propose for inclusion in the standard library. (Nose has this
feature as well, but I doubt their implementation would be usable,
since their idea of a 'test' isn't really TestCase based.)
testresources, by contrast, is a tested, existing package, which
people are already using, using a long-standing integration mechanism
that has been part of unittest since its first implementation.
Granted, I would not contest that it is "immature"; it is still
fairly new, and doesn't have a huge number of uses, but it's odd to
criticize it on grounds of maturity when it's so much *more* mature
than the alternative.
While superficially the programming interface to testresources is
slightly more unusual, this is only because programmers don't think to
hard about what unittest actually does with your code, and
testresources requires a little more familiarity with that.
And setUpClass does inevitably start to break those integration
points down, because it implies certain things, like the fact that
classes and modules are suites, or are otherwise grouped together in
test ordering.
I expect that is what the majority of unittest users already believe.
Yes, but they're wrong, and enforcing this misconception doesn't help
anyone. There are all kinds of assumptions that most python
developers have about how Python works which are vaguely incorrect
abstractions over the actual behavior.
This makes it difficult to create custom suites, to do custom
ordering, custom per-test behavior (like metrics collection before
and after run(), or gc.collect() after each test, or looking for
newly-opened-but-not-cleaned-up external resources like file
descriptors after each tearDown).
True, the list never ends.
Again: these are all concrete features that *users* of test
frameworks want, not just idle architectural fantasy of us framework
hackers.
I expect that most bleeding edge users will end up writing a custom
framework, or at least work with a bleeding edge framework that change
change rapidly to meet their urgent needs.
Yes, exactly. Users who want an esoteric feature like setUpClass
should use a custom framework :). The standard library's job, in my
view, is to provide an integration point for those more advanced
framework so that different "bleeding edge" frameworks have a
mechanism to communicate, and users have some level of choice about
what tools they use to run their tests.
I have not always thought this way. Originally, my intention was for
twisted's test framework to be a complete departure from the standard
library and just do its own thing. But, both users of the framework
and more savvy test developers have gradually convinced me that it's
really useful to be able to load stdlib tests with a variety of tools.
This doesn't mean that I think the stdlib should stop changing
completely, but I do think changes which potentially break
compatibility with _basic_ test framework things like re-ordering
tests or overriding a core method like 'run' need to be done extremely
carefully.
I haven't had the opportunity to read the entire thread, so I don't
know if this discussion has come to fruition, but I can see that
some attention has been paid to these difficulties. I have no
problem with setUpClass or tearDownClass hooks *per se*, as long as
they can be implemented in a way which explicitly preserves
extensibility.
That's good to know. I have no doubt they (and setUpModule c.s.) can
be done in a clean, extensible way. And that doesn't mean we couldn't
also add other features -- after all, not all users have the same
needs. (If you read the Zen of Python, you'll see that TOOWTDI has
several qualifications. :-)
Recent messages indicate that this is still a problem. But perhaps it
will be solved! I can yell at Michael in person in a few days, anyway :).
<snip a bunch of questions about behavior of the feature>
All those answers were pretty reasonable, so I will avoid retreading them.
testresources very neatly sidesteps this problem by just providing
an API to say "this test case depends on that test resource",
without relying on the grouping of tests within classes, modules, or
packages. Of course you can just define a class-level or
module-level resource and then have all your tests depend on it,
which gives you the behavior of setUpClass and setUpModule in a more
general way.
I wish it was always a matter of "resources". I've seen use cases for
module-level setup that were much messier than that (e.g. fixing
import paths). I expect it will be a while before the testresources
design has been shaken out sufficiently for it to be included in the
stdlib.
Why isn't loadable code a "resource" like anything else? I haven't
used testresources specifically for this, but I've definitely written
fixture setup and teardown code that dealt with sys.path.
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your
employer, to release me from all obligations and waivers arising from any and all
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS
AGREEMENTS") that I have entered into with your employer, its partners, licensors,
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me from any BOGUS AGREEMENTS
on behalf of your employer.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
