Terry J. Reedy <tjre...@udel.edu> added the comment:

This issue confuses two different testing issues.  First is selection of test 
data before the test versus generation while testing (usually 'randomly')  
Second is how the function is tested.  All tests test some property of the 
inputs and function, but it is useful to distinguish between testing that a 
function produces a specific expected output versus testing something more 
abstract or defined in terms of the inputs.

Example 1: 2 + 3 = 5 (a + b = (known) c) versus 2 + 3 = 3 + 2 (a + b = b + a) 
and (2 + 3) + 4 = 2 + (3 + 4).

Example 2: sorted([3,5,1,4]) = [1,3,4,5] versus is_sorted(sorted([random 

The latter are what hypothesis people mean by 'property tests'.  Such property 
tests can be done with either fixed or random pairs in unittest module tests.  
The test that a module compiles is a weak property test, but better than 
nothing.  A test that that all lines of code runs at least once without an 
unexpected exception a stronger property test.  I consider it roughly on a par 
with min(s) <= mean(s) <= max(s).

The connection between randomized input testing, such as with hypothesis, and 
property tests is that with random inputs, one cannot test f(input) against an 
'expected' value unless one has an alternate means of producing the expected 
value.  It is often easier to test a parameterized equation or other properties.

The problem with random input tests in not that they are 'flakey', but that 
they are useless unless someone is going to pay attention to failures and try 
to find the cause.  This touches on the difference between regression testing 
and bug-finding tests.  CPython CI is the former, and marred at that by buggy 
randomly failing tests.

My conclusion: bug testing would likely be a good idea, but should be done 
separate from the CI test suite.  Such testing should only be done for modules 
with an active maintainer who would welcome failure reports.


Python tracker <rep...@bugs.python.org>
Python-bugs-list mailing list

Reply via email to