On Fri, 26 Aug 2022 at 15:55, Jean-Paul Calderone <exar...@twistedmatrix.com> wrote: > > Hello all, > > I'd like to start a discussion (but hopefully a brief one!) about using > property testing techniques in Twisted. > > For those who aren't familiar with them, a property test is a kind of unit > test written such that it can assert a certain invariant holds over a range > of inputs. This is in contrast with more conventional unit tests in which a > single test asserts a certain invariant holds for a single input and > typically many such tests are written to cover the necessary range of inputs. > > Twisted mostly uses conventional unit tests though it does include a few > property tests written in an ad hoc way. If you search for loops in the test > suite you quickly come across them. For example, > twisted.test.test_amp.ParsingTests includes several. test_booleanValues does > not use a loop but it is a property test. test_ParsingRoundTrip does use a > loop and is also a property test. In these cases, the developer knew they > wanted to exercise the implementation against more than one input but they > were limited in how exhaustive they could be by the tools available. > > There is a popular library for property testing in Python called Hypothesis. > I've pushed some changes to a branch to demonstrate usage (note: not > currently for review) of some of the basic APIs and the tests that result: > > The given decorator marks a test as a property test. The consequence of this > is usually that the test method runs more than once. The "strategies" passed > to given determine how it is called. > Strategies like tuples, lists, integers specify what values are passed to the > decorated function. Each will correspond to a parameter the test method > accepts. A simple strategy like integers will result in int values being > passed. Many strategies can be composed. A strategy like tuples will result > in tuple values being passed, with the elements of the value determined by > the strategies passed to the tuples strategy. > Helper functions like assume support dealing with corner cases related to > invalid inputs which are tricky to avoid in a strategy. If it is hard to > constructively define exactly the right set of inputs, you can define a > strategy that finds "too many" and filter out the bad ones afterwards. > > The test is then written to make an assertion that is correct with respect to > the specific values the test is called with. For complex code, it can take a > while to get used to writing tests in this style but for simpler code it's > often only a matter of using the strategy-provided values instead of some > hard-coded values. > > Once using given, the test method actually runs many times, each time with > different values "drawn" from the given strategies. When the test fails, > Hypothesis tries to "shrink" the values - it simplifies them as far as it can > while still causing the test to fail. When it has simplified them as much as > it can without making the test pass, those simpler values are reported. This > means that Hypothesis can find complex inputs that break the code under test > but when simple values will do, those are the ones a developer gets to work > with. > > There are many other benefits to this style of testing, such as reduced > effort required to write comprehensive tests and improved code re-use in the > test suite. I won't describe them further here but you can learn more at the > Hypothesis website - https://hypothesis.works/. > > I think property testing is a valuable tool in the testing toolbox and > Twisted should consider adopting Hypothesis to employ it for some tests. > > Thoughts? > > Jean-Paul
Hi Jean-Paul Thanks for the info. I am +1 for using hypothesis. My initial reaction to hypothesis and hamcrest and attrs was -1 :) The reason was that it requires extra knowledge to contribute to Twisted. At the same time, I think that hypothesis and attrs should be considered kind of "stdlib" or best practice. So if you are Python dev, you should learn them :) -- Adi Roiban _______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-le...@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/4NJZDV7TEPH7CD5O4IOFIXSY7LE6NWFO/ Code of Conduct: https://twisted.org/conduct
