Please do, if You find it useful. /Anders
On Thu, Mar 24, 2011 at 8:26 AM, Torben Hoffmann <[email protected]> wrote: > Hi Anders, > > That is a very good description - I will integrate that into the explanation > if you don't mind. > > Cheers, > Torben > > On Thu, Mar 24, 2011 at 15:20, Anders Nygren <[email protected]> > wrote: >> >> Hi >> The way quickcheck and triq, and I assume proper, does it, is that >> - tests are run on s sequence of random generated values, as defined by >> the >> generators. >> - when a test fails, it starts shrinking the test data but simplifying >> the the data >> that caused the failure, e.g lists get shorter or integers smaller >> until the test >> does not fail anymore >> - the smallest/simplest failing data set is reported and SAVED, so it easy >> to >> rerun the test with the failing data after a fix has been done in >> order to verify that >> the fix actually solves the problem. >> >> /Anders >> >> On Thu, Mar 24, 2011 at 8:10 AM, Eric Merritt <[email protected]> >> wrote: >> > Torben, >> > >> > Great first stab I think, and I suspect its going to turn into a good >> > proper article. I am really curious about narrowing, or specifically >> > reproducible failures. >> > >> > The impression I get is that the generated values are generated, that >> > is there is some level of randomness in the generation. If that is the >> > case, how do you make property test failures reproducible? I also >> > suspect this is where narrowing comes in to play. >> > >> > Eric >> > >> > On Wed, Mar 23, 2011 at 5:14 PM, Torben Hoffmann >> > <[email protected]> wrote: >> >> Hi, >> >> >> >> I have started on my explantion of what property based testing is - it >> >> is >> >> not done yet, but I thought I would release early and get some >> >> feed-back so >> >> I can avoid having text that requires too much inside knowledge. >> >> >> >> Please comment and provide alternative descriptions - the goal is to >> >> get a >> >> text that will help people that wants to get into property based >> >> testing and >> >> once you have learned it it will be so much more difficult for you to >> >> spot >> >> insider information. >> >> >> >> Cheers, >> >> Torben >> >> >> >> >> >> ---------------------------------------------------------------------------------------------------------------------------- >> >> Property based testing for unit testers >> >> >> >> The purpose of the short document is to help people who are familiar >> >> with >> >> unit testing understand how property based testing (PBT) differs, but >> >> also >> >> where the thinking is the same. >> >> >> >> This document focusses on the PBT tool PropEr for Erlang since that is >> >> what >> >> I am familiar with, but the general principles applies to all PBT tools >> >> regardless of which language they are written in. >> >> >> >> The approach taken here is that we hear from people who are used to >> >> working >> >> with unit testing regarding how they think when designing their tests >> >> and >> >> how a concrete test might look. >> >> >> >> These descriptions are then "converted" into the way it works with PBT, >> >> with >> >> a clear focus on what stays the same and what is different. >> >> >> >> Testing philosophies >> >> >> >> A quote from Martin Logan: >> >> >> >> For me unit testing is about contracts. I think about the same things >> >> I think about when I write statements like {ok, Resp} = >> >> Mod:Func(Args). Unit testing and writing specs are very close for me. >> >> Hypothetically speaking lets say a function should return return {ok, >> >> string()} | {error, term()} for all given input parameters then my >> >> unit tests should be able to show that for a representative set of >> >> input parameters that those contracts are honored. The art comes in >> >> thinking about what that set is. >> >> >> >> The trap in writing all your own tests can often be that we think >> >> about the set in terms of what we coded for and not what may indeed be >> >> asked of our function. As the code is tried in further exploratory >> >> testing >> >> and >> >> in production new input parameter sets for which the given function >> >> does not >> >> meet >> >> the stated contract are discovered and added to the test case once a >> >> fix has been put into place. >> >> This is a very good description of what the ground rules for unit >> >> testing >> >> are: >> >> >> >> Checking that contracts are obeyed. >> >> Creating a representative set of input parameters. >> >> >> >> The former is very much part of PBT - each property you write will >> >> check a >> >> contract, so that thinking is the same. Note: the word property is used >> >> instead of test case since you the property says something about your >> >> system >> >> in general and not just a single run of things as with a test case. >> >> More on >> >> this right away! >> >> >> >> What PBT does different is the input parameters. Instead of crafting a >> >> few >> >> off-the-top-of-your head input parameters you specify the rules for >> >> creating >> >> the input parameters and let your property based testing tool generate >> >> the >> >> input parameters. >> >> >> >> The functions you write to specify the rules are called generators. >> >> >> >> Suppose you want to test a sorting function that takes a list of >> >> integers >> >> and returns the list sorted. >> >> The generator for this would be a function that returns a list with a >> >> random >> >> number of elements, each of which is a random number. >> >> >> >> One can argue that that sort of randomness can be applied to unit >> >> testing as >> >> well and indeed it can. It is just easier with a good PBT tool since it >> >> has >> >> facilities that allows you to state the above generator as: >> >> list(integer()) >> >> >> >> But it does not stop at generation of input parameters. If you have >> >> more >> >> complex tests where you have to generate a series of events and keep >> >> track >> >> of some state then your PBT tool will generate random sequences of >> >> events >> >> which corresponds to legal sequences of events and test that your >> >> system >> >> behaves correctly for all sequences. >> >> >> >> So when you have written a property with associated generators you have >> >> in >> >> fact created something that can create numerous test cases - you just >> >> have >> >> to tell your PBT tool how many test cases you want to check the >> >> property on. >> >> >> >> Raising the bar by shrinking >> >> >> >> At this point you might still have the feeling that introducing the >> >> notion >> >> of some sort of generators to your unit testing tool of choice would >> >> bring >> >> you on par with PBT tools, but wait there is more to come. >> >> >> >> When a PBT tool creates a test case that fails there is real chance >> >> that it >> >> has created a long test case or some big input parameters - trying to >> >> debug >> >> that is very much like receiving a humongous log from a system in the >> >> field >> >> and try to figure out what cause the system to fail. >> >> >> >> Enter shrinking... >> >> >> >> When a test case fails the PBT tool will try to shrink the failing test >> >> case >> >> down to the essentials by stripping out input elements or events that >> >> does >> >> not cause the failure. In most cases this results in a very short >> >> counterexample that clearly states which events and inputs are required >> >> to >> >> break a property. >> >> >> >> As we go through some concrete examples later the effects of shrinking >> >> will >> >> be shown. >> >> >> >> ---------------------------------------------------------------------------------------------------------------------------- >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> http://www.linkedin.com/in/torbenhoffmann >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "erlware-dev" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/erlware-dev?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "erlware-dev" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/erlware-dev?hl=en. >> > >> > > > > > -- > http://www.linkedin.com/in/torbenhoffmann > -- You received this message because you are subscribed to the Google Groups "erlware-dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/erlware-dev?hl=en.
