On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote:
> I think Steven's is backwards in its own way.
>
> - Contracts test the space of arguments *actually used during testing
> period* (or during initial production if the performance hit is
> acceptable).
> - Unit tests test the space of arguments *thought of by the developers*.
>
> *A priori,* either one of those can cover cases not addressed by the
> other.
Fair point.
But given that in general unit tests tend to only exercise a handful of
values (have a look at the tests in the Python stdlib) I think it is
fair to say that in practice unit tests typically do not have anywhere
near the coverage of live data used during alpha and beta testing.
> If unit tests use the hypothesis library or similar approaches,
> unit tests might very well examine arguments unlikely to be encountered in
> real-world (or test phase) use...
Indeed.
We can consider all of these things as complementary:
- doctests give us confidence that the documentation hasn't rotted;
- unit tests give us confidence that corner cases are tested;
- contracts give us confidence that regular and common cases are tested;
- regression tests give us confidence that bugs aren't re-introduced;
- smoke tests give us confidence that the software at least will run;
- static type checking allows us to drop type checks from our unit
tests and contracts;
but of course there can be overlap. And that's perfectly fine.
[...]
> - Half of the checks are very far away, in a separate file, assuming
> > I even remembered or bothered to write the test.
> >
>
> To me, this is the GREATEST VIRTUE of unit tests over DbC. It puts the
> tests far away where they don't distract from reading and understanding the
> function itself. I rarely want my checks proximate since I wear a very
> different hat when thinking about checks than when writing functionality
> (ideally, a lot of the time, I wear the unit test hat *before* I write the
> implementation; TDD is usually good practice).
I'm curious. When you write a function or method, do you include input
checks? Here's an example from the Python stdlib (docstring removed for
brevity):
# bisect.py
def insort_right(a, x, lo=0, hi=None):
if lo < 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi)//2
if x < a[mid]: hi = mid
else: lo = mid+1
a.insert(lo, x)
Do you consider that check for lo < 0 to be disruptive? How would you
put that in a unit test?
That check is effectively a pre-condition. Putting aside the question of
which exception should be raised (AssertionError or ValueError), we
could re-write that as a contract:
def insort_right(a, x, lo=0, hi=None):
require:
lo >= 0
# implementation follows, as above
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi)//2
if x < a[mid]: hi = mid
else: lo = mid+1
a.insert(lo, x)
Do you consider that precondition check for lo >= to be disruptive? More
or less disruptive than when it was in the body of the function
implementation?
> > - The post-conditions aren't checked unless I run my test suite, and
> > then they only check the canned input in the test suite.
> >
>
> Yes, this is a great advantage of unit tests. No cost until you explicitly
> run them.
If you're worried about the cost of verifying your program does the
right thing during testing and development, I think you're doing
something wrong :-)
If there are specific functions/classes where the tests are insanely
expensive, that's one thing. I have some code that wants to verify that
a number is prime as part of an informal post-condition check, but if it
is a *big* prime that check is too costly so I skip it.
But in general, if I'm testing or under active development, what do I
care if the program takes 3 seconds to run instead of 2.5 seconds?
Either way, its finished by the time I come back from making my coffee
:-)
But more seriously, fine, if a particular contract is too expensive to
run, disable it or remove it and add some unit tests. And then your
devs will complain that the unit tests are too slow, and stop running
them, and that's why we can't have nice things... *wink*
--
Steve
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
