[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Zac Hatfield-Dodds added the comment: I've recently had [1] reported, which makes https://bugs.python.org/issue45738 the *third* parser bug [2] that Hypothesmith caught after release, and the second in a stable version - so I'll be maintaining a workaround for some time. I remain happy to

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Change by Aaron Meurer : -- nosy: +asmeurer ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Change by Brandt Bucher : -- nosy: +brandtbucher ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Terry J. Reedy added the comment: In order to understand what Paul is concretely proposing, I read and partly reviewed his PR. I thought about maintainability. Part A adds test.support.hypothesis_helper, similar in purpose to other specialized xyz_helper modules. Hypothesis_helper imports

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Paul Ganssle added the comment: > I use hypothesis during development, but don't have a need for in the the > standard library. By the time code lands there, we normally have a specific > idea of what edge cases needs to be in the tests. The suggestion I've made here is that we use @example

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Raymond Hettinger added the comment: Here's my two cents worth on the subject. * I use hypothesis during development, but don't have a need for in the the standard library. By the time code lands there, we normally have a specific idea of what edge cases needs to be in the tests. * For

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: I withdraw my objection against Hypothetlsis per se. Others should debate whether the 3rd party dependency is okay. -- --Guido (mobile) -- ___ Python tracker

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: Having read the parts of the paper that explain shortening, things are making more sense now. I have also successfully contributed type annotations to minithesis! :-) -- ___ Python tracker

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: It is important to me, because I don't like having a dependency on third-party code that is a black box. Remember that in the end it's not me you have to convince but the SC. -- ___ Python tracker

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Paul Ganssle added the comment: I do not want to dissuade you from figuring out how minithesis / hypothesis works (far from it), but I'm wondering if the question of how shrinking works is germane to the issue at hand, which is whether or not hypothesis / property-based-testing is suitable

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: Few things incite me quite as much as being told that I'm not supposed to understand something, so now I'm actually making an effort to follow the ECOOP paper. Honestly, it's not too bad (a lot more readable than type theory papers :-). I'll get back to

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Zac Hatfield-Dodds added the comment: > Okay, well, I'm trying to understand minithesis.py, but I am despairing. The > shrinking code (really everything in class TestingState) is too much to grok. > ... I suppose I have to read the paper at >

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: Okay, well, I'm trying to understand minithesis.py, but I am despairing. The shrinking code (really everything in class TestingState) is too much to grok. You can tell from the amount of comments that this is the tour-de-force of the code, but the

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Paul Ganssle added the comment: @Terry > The problem with random input tests in not that they are 'flakey', but that > they are useless unless someone is going to pay attention to failures and try > to find the cause. This touches on the difference between regression testing > and

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Terry J. Reedy added the comment: This issue confuses two different testing issues. First is selection of test data before the test versus generation while testing (usually 'randomly') Second is how the function is tested. All tests test some property of the inputs and function, but it

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: Thanks, I will look into that before stirring up more dust. -- nosy: +Guido.van.Rossum ___ Python tracker ___

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Zac Hatfield-Dodds added the comment: (I think structuring this as a high-level explanation is clearer than point-by-point replies - it covers most of them, but the logic is hopefully easier to follow) The core idea of Hypothesis is that making random choices is equivalent to parsing a

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Guido van Rossum added the comment: I would like to have a more thorough discussion about the desirability of using Hypothesis first, since I feel that there is a rather hard "sell" going on. I brought this up in the SC tracker (https://github.com/python/steering-council/issues/65) but I

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Change by Paul Ganssle : -- keywords: +patch pull_requests: +21805 pull_request: https://github.com/python/cpython/pull/22863 ___ Python tracker ___

[issue42109] Use hypothesis for testing the standard library, falling back to stubs

New submission from Paul Ganssle : Following up on this year's language summit, I would like to propose that we start integrating property tests into the standard library. Zac Hatfield-Dodds (along with myself and Carl Friedrich Bolz-Tereick) have put together this repository of tests that