Torben, this is really nice start in outlining what you will be talking about. I suspect you will already be doing this but adding a few more examples on how properties might be specified and how they might be judged to be correct would be great. For example how to I specifiy that the result of the random numbers should be sorted and how do I prove that with a generic spec that can also be used to prove other more complex properties.
oh, and one quick typo "case since you the property says something" I am looking forward to reading more. Cheers, Martin On Wed, Mar 23, 2011 at 5:14 PM, Torben Hoffmann <[email protected]> wrote: > Hi, > > I have started on my explantion of what property based testing is - it is > not done yet, but I thought I would release early and get some feed-back so > I can avoid having text that requires too much inside knowledge. > > Please comment and provide alternative descriptions - the goal is to get a > text that will help people that wants to get into property based testing and > once you have learned it it will be so much more difficult for you to spot > insider information. > > Cheers, > Torben > > ---------------------------------------------------------------------------------------------------------------------------- > Property based testing for unit testers > > The purpose of the short document is to help people who are familiar with > unit testing understand how property based testing (PBT) differs, but also > where the thinking is the same. > > This document focusses on the PBT tool PropEr for Erlang since that is what > I am familiar with, but the general principles applies to all PBT tools > regardless of which language they are written in. > > The approach taken here is that we hear from people who are used to working > with unit testing regarding how they think when designing their tests and > how a concrete test might look. > > These descriptions are then "converted" into the way it works with PBT, with > a clear focus on what stays the same and what is different. > > Testing philosophies > > A quote from Martin Logan: > > For me unit testing is about contracts. I think about the same things > I think about when I write statements like {ok, Resp} = > Mod:Func(Args). Unit testing and writing specs are very close for me. > Hypothetically speaking lets say a function should return return {ok, > string()} | {error, term()} for all given input parameters then my > unit tests should be able to show that for a representative set of > input parameters that those contracts are honored. The art comes in > thinking about what that set is. > > The trap in writing all your own tests can often be that we think > about the set in terms of what we coded for and not what may indeed be > asked of our function. As the code is tried in further exploratory testing > and > in production new input parameter sets for which the given function does not > meet > the stated contract are discovered and added to the test case once a > fix has been put into place. > This is a very good description of what the ground rules for unit testing > are: > > Checking that contracts are obeyed. > Creating a representative set of input parameters. > > The former is very much part of PBT - each property you write will check a > contract, so that thinking is the same. Note: the word property is used > instead of test case since you the property says something about your system > in general and not just a single run of things as with a test case. More on > this right away! > > What PBT does different is the input parameters. Instead of crafting a few > off-the-top-of-your head input parameters you specify the rules for creating > the input parameters and let your property based testing tool generate the > input parameters. > > The functions you write to specify the rules are called generators. > > Suppose you want to test a sorting function that takes a list of integers > and returns the list sorted. > The generator for this would be a function that returns a list with a random > number of elements, each of which is a random number. > > One can argue that that sort of randomness can be applied to unit testing as > well and indeed it can. It is just easier with a good PBT tool since it has > facilities that allows you to state the above generator as: > list(integer()) > > But it does not stop at generation of input parameters. If you have more > complex tests where you have to generate a series of events and keep track > of some state then your PBT tool will generate random sequences of events > which corresponds to legal sequences of events and test that your system > behaves correctly for all sequences. > > So when you have written a property with associated generators you have in > fact created something that can create numerous test cases - you just have > to tell your PBT tool how many test cases you want to check the property on. > > Raising the bar by shrinking > > At this point you might still have the feeling that introducing the notion > of some sort of generators to your unit testing tool of choice would bring > you on par with PBT tools, but wait there is more to come. > > When a PBT tool creates a test case that fails there is real chance that it > has created a long test case or some big input parameters - trying to debug > that is very much like receiving a humongous log from a system in the field > and try to figure out what cause the system to fail. > > Enter shrinking... > > When a test case fails the PBT tool will try to shrink the failing test case > down to the essentials by stripping out input elements or events that does > not cause the failure. In most cases this results in a very short > counterexample that clearly states which events and inputs are required to > break a property. > > As we go through some concrete examples later the effects of shrinking will > be shown. > ---------------------------------------------------------------------------------------------------------------------------- > > > > > > > -- > http://www.linkedin.com/in/torbenhoffmann > > -- > You received this message because you are subscribed to the Google Groups > "erlware-dev" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/erlware-dev?hl=en. > -- Martin Logan Erlang & OTP in Action (Manning) http://manning.com/logan http://twitter.com/martinjlogan http://erlware.org -- You received this message because you are subscribed to the Google Groups "erlware-dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/erlware-dev?hl=en.
