Hey Damian!
Wasn't sure if follow up privately or to the list, sending this to the
list for now.
On Thu, 16 Feb 2012 22:09:44 +0100, Damian Johnson <[email protected]>
wrote:
Nice posts and thanks for improving the tor tests! I'm not entirely in
agreement with the last point (about tests covering all cases). If the
test space is decently small then exercising everything can better
ensure that you don't violate a set of invariants. For example, one of
stem's unit tests attempt every combination of authentication methods
against every set of failures they can encounter, making sure that we
properly report success/failure and never raise an unexpected type of
exception...
https://gitweb.torproject.org/stem.git/blob/HEAD:/test/unit/connection/authentication.py#l69
But that said, in general you're probably right. -Damian
Thanks for the feedback. You're right, covering all cases is not bad in
itself. The last point was the weakest, both because I had thought the
least about it, and because I didn't have an example to back it up.
The main thing I had in mind when I wrote that point was some test
cases I had recently reviewed at work. They make some statistical
computations, and for the tests it *creates* hundreds of test cases (and
calculates the expected values!) in a loop, then checks the actual values
are the same as the (calculated) expected. That makes the test code as
complex as the production code, making it very hard to know if the tests
are correct, and once something fails, they're pretty hard to debug.
As I had seen it before, and my impression was that the reason for this
happening was developers not realising test code is different than
"production" code (among other things, because it doesn't have to be
"generic" as in "covering all cases"). In hindsight, after you comment,
maybe the main/only bad thing about that test was *calculating* the
expected value.
--
Esteban
_______________________________________________
tor-dev mailing list
[email protected]
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
