On Thu, Mar 24, 2011 at 17:12, Jordan Wilberding <[email protected]>wrote:
> Can we use a monte carlo model, where we can guarantee things to be > failure-free with a certain amount of probability? > > I don't think so, but I am not an expert on Monte Carlo models! Could you explain how a MC model can be used to predict failure-free probabilities? Normally you would use MC simulation to get a prediction of a range, e.g., when you are modelling when a series of tasks with given distribution will be done if executed right after one another. What you can do with most PBT tools is to collect code coverage numbers while you test as well as statistics on what sort of tests has been generated. Is that in the direction you were thinking of? Cheers, Torben > Thanks! > Jordan Wilberding > > > On Thu, Mar 24, 2011 at 10:10 AM, Eric Merritt <[email protected]>wrote: > >> Torben, >> >> Great first stab I think, and I suspect its going to turn into a good >> proper article. I am really curious about narrowing, or specifically >> reproducible failures. >> >> The impression I get is that the generated values are generated, that >> is there is some level of randomness in the generation. If that is the >> case, how do you make property test failures reproducible? I also >> suspect this is where narrowing comes in to play. >> >> Eric >> >> On Wed, Mar 23, 2011 at 5:14 PM, Torben Hoffmann >> <[email protected]> wrote: >> > Hi, >> > >> > I have started on my explantion of what property based testing is - it >> is >> > not done yet, but I thought I would release early and get some feed-back >> so >> > I can avoid having text that requires too much inside knowledge. >> > >> > Please comment and provide alternative descriptions - the goal is to get >> a >> > text that will help people that wants to get into property based testing >> and >> > once you have learned it it will be so much more difficult for you to >> spot >> > insider information. >> > >> > Cheers, >> > Torben >> > >> > >> ---------------------------------------------------------------------------------------------------------------------------- >> > Property based testing for unit testers >> > >> > The purpose of the short document is to help people who are familiar >> with >> > unit testing understand how property based testing (PBT) differs, but >> also >> > where the thinking is the same. >> > >> > This document focusses on the PBT tool PropEr for Erlang since that is >> what >> > I am familiar with, but the general principles applies to all PBT tools >> > regardless of which language they are written in. >> > >> > The approach taken here is that we hear from people who are used to >> working >> > with unit testing regarding how they think when designing their tests >> and >> > how a concrete test might look. >> > >> > These descriptions are then "converted" into the way it works with PBT, >> with >> > a clear focus on what stays the same and what is different. >> > >> > Testing philosophies >> > >> > A quote from Martin Logan: >> > >> > For me unit testing is about contracts. I think about the same things >> > I think about when I write statements like {ok, Resp} = >> > Mod:Func(Args). Unit testing and writing specs are very close for me. >> > Hypothetically speaking lets say a function should return return {ok, >> > string()} | {error, term()} for all given input parameters then my >> > unit tests should be able to show that for a representative set of >> > input parameters that those contracts are honored. The art comes in >> > thinking about what that set is. >> > >> > The trap in writing all your own tests can often be that we think >> > about the set in terms of what we coded for and not what may indeed be >> > asked of our function. As the code is tried in further exploratory >> testing >> > and >> > in production new input parameter sets for which the given function does >> not >> > meet >> > the stated contract are discovered and added to the test case once a >> > fix has been put into place. >> > This is a very good description of what the ground rules for unit >> testing >> > are: >> > >> > Checking that contracts are obeyed. >> > Creating a representative set of input parameters. >> > >> > The former is very much part of PBT - each property you write will check >> a >> > contract, so that thinking is the same. Note: the word property is used >> > instead of test case since you the property says something about your >> system >> > in general and not just a single run of things as with a test case. More >> on >> > this right away! >> > >> > What PBT does different is the input parameters. Instead of crafting a >> few >> > off-the-top-of-your head input parameters you specify the rules for >> creating >> > the input parameters and let your property based testing tool generate >> the >> > input parameters. >> > >> > The functions you write to specify the rules are called generators. >> > >> > Suppose you want to test a sorting function that takes a list of >> integers >> > and returns the list sorted. >> > The generator for this would be a function that returns a list with a >> random >> > number of elements, each of which is a random number. >> > >> > One can argue that that sort of randomness can be applied to unit >> testing as >> > well and indeed it can. It is just easier with a good PBT tool since it >> has >> > facilities that allows you to state the above generator as: >> > list(integer()) >> > >> > But it does not stop at generation of input parameters. If you have more >> > complex tests where you have to generate a series of events and keep >> track >> > of some state then your PBT tool will generate random sequences of >> events >> > which corresponds to legal sequences of events and test that your system >> > behaves correctly for all sequences. >> > >> > So when you have written a property with associated generators you have >> in >> > fact created something that can create numerous test cases - you just >> have >> > to tell your PBT tool how many test cases you want to check the property >> on. >> > >> > Raising the bar by shrinking >> > >> > At this point you might still have the feeling that introducing the >> notion >> > of some sort of generators to your unit testing tool of choice would >> bring >> > you on par with PBT tools, but wait there is more to come. >> > >> > When a PBT tool creates a test case that fails there is real chance that >> it >> > has created a long test case or some big input parameters - trying to >> debug >> > that is very much like receiving a humongous log from a system in the >> field >> > and try to figure out what cause the system to fail. >> > >> > Enter shrinking... >> > >> > When a test case fails the PBT tool will try to shrink the failing test >> case >> > down to the essentials by stripping out input elements or events that >> does >> > not cause the failure. In most cases this results in a very short >> > counterexample that clearly states which events and inputs are required >> to >> > break a property. >> > >> > As we go through some concrete examples later the effects of shrinking >> will >> > be shown. >> > >> ---------------------------------------------------------------------------------------------------------------------------- >> > >> > >> > >> > >> > >> > >> > -- >> > http://www.linkedin.com/in/torbenhoffmann >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "erlware-dev" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/erlware-dev?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "erlware-dev" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/erlware-dev?hl=en. >> >> > -- http://www.linkedin.com/in/torbenhoffmann -- You received this message because you are subscribed to the Google Groups "erlware-dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/erlware-dev?hl=en.
