I would also be interested in how you would do this Jordan, I am not sure I grock it (actually I am sure I don't)
On Thu, Mar 24, 2011 at 3:28 PM, Torben Hoffmann <[email protected]> wrote: > On Thu, Mar 24, 2011 at 17:12, Jordan Wilberding <[email protected]> > wrote: >> >> Can we use a monte carlo model, where we can guarantee things to be >> failure-free with a certain amount of probability? > > I don't think so, but I am not an expert on Monte Carlo models! > > Could you explain how a MC model can be used to predict failure-free > probabilities? > Normally you would use MC simulation to get a prediction of a range, e.g., > when you are modelling when a series of tasks with given distribution will > be done if executed right after one another. > > What you can do with most PBT tools is to collect code coverage numbers > while you test as well as statistics on what sort of tests has been > generated. > Is that in the direction you were thinking of? > > Cheers, > Torben > > >> >> Thanks! >> Jordan Wilberding >> >> On Thu, Mar 24, 2011 at 10:10 AM, Eric Merritt <[email protected]> >> wrote: >>> >>> Torben, >>> >>> Great first stab I think, and I suspect its going to turn into a good >>> proper article. I am really curious about narrowing, or specifically >>> reproducible failures. >>> >>> The impression I get is that the generated values are generated, that >>> is there is some level of randomness in the generation. If that is the >>> case, how do you make property test failures reproducible? I also >>> suspect this is where narrowing comes in to play. >>> >>> Eric >>> >>> On Wed, Mar 23, 2011 at 5:14 PM, Torben Hoffmann >>> <[email protected]> wrote: >>> > Hi, >>> > >>> > I have started on my explantion of what property based testing is - it >>> > is >>> > not done yet, but I thought I would release early and get some >>> > feed-back so >>> > I can avoid having text that requires too much inside knowledge. >>> > >>> > Please comment and provide alternative descriptions - the goal is to >>> > get a >>> > text that will help people that wants to get into property based >>> > testing and >>> > once you have learned it it will be so much more difficult for you to >>> > spot >>> > insider information. >>> > >>> > Cheers, >>> > Torben >>> > >>> > >>> > ---------------------------------------------------------------------------------------------------------------------------- >>> > Property based testing for unit testers >>> > >>> > The purpose of the short document is to help people who are familiar >>> > with >>> > unit testing understand how property based testing (PBT) differs, but >>> > also >>> > where the thinking is the same. >>> > >>> > This document focusses on the PBT tool PropEr for Erlang since that is >>> > what >>> > I am familiar with, but the general principles applies to all PBT tools >>> > regardless of which language they are written in. >>> > >>> > The approach taken here is that we hear from people who are used to >>> > working >>> > with unit testing regarding how they think when designing their tests >>> > and >>> > how a concrete test might look. >>> > >>> > These descriptions are then "converted" into the way it works with PBT, >>> > with >>> > a clear focus on what stays the same and what is different. >>> > >>> > Testing philosophies >>> > >>> > A quote from Martin Logan: >>> > >>> > For me unit testing is about contracts. I think about the same things >>> > I think about when I write statements like {ok, Resp} = >>> > Mod:Func(Args). Unit testing and writing specs are very close for me. >>> > Hypothetically speaking lets say a function should return return {ok, >>> > string()} | {error, term()} for all given input parameters then my >>> > unit tests should be able to show that for a representative set of >>> > input parameters that those contracts are honored. The art comes in >>> > thinking about what that set is. >>> > >>> > The trap in writing all your own tests can often be that we think >>> > about the set in terms of what we coded for and not what may indeed be >>> > asked of our function. As the code is tried in further exploratory >>> > testing >>> > and >>> > in production new input parameter sets for which the given function >>> > does not >>> > meet >>> > the stated contract are discovered and added to the test case once a >>> > fix has been put into place. >>> > This is a very good description of what the ground rules for unit >>> > testing >>> > are: >>> > >>> > Checking that contracts are obeyed. >>> > Creating a representative set of input parameters. >>> > >>> > The former is very much part of PBT - each property you write will >>> > check a >>> > contract, so that thinking is the same. Note: the word property is used >>> > instead of test case since you the property says something about your >>> > system >>> > in general and not just a single run of things as with a test case. >>> > More on >>> > this right away! >>> > >>> > What PBT does different is the input parameters. Instead of crafting a >>> > few >>> > off-the-top-of-your head input parameters you specify the rules for >>> > creating >>> > the input parameters and let your property based testing tool generate >>> > the >>> > input parameters. >>> > >>> > The functions you write to specify the rules are called generators. >>> > >>> > Suppose you want to test a sorting function that takes a list of >>> > integers >>> > and returns the list sorted. >>> > The generator for this would be a function that returns a list with a >>> > random >>> > number of elements, each of which is a random number. >>> > >>> > One can argue that that sort of randomness can be applied to unit >>> > testing as >>> > well and indeed it can. It is just easier with a good PBT tool since it >>> > has >>> > facilities that allows you to state the above generator as: >>> > list(integer()) >>> > >>> > But it does not stop at generation of input parameters. If you have >>> > more >>> > complex tests where you have to generate a series of events and keep >>> > track >>> > of some state then your PBT tool will generate random sequences of >>> > events >>> > which corresponds to legal sequences of events and test that your >>> > system >>> > behaves correctly for all sequences. >>> > >>> > So when you have written a property with associated generators you have >>> > in >>> > fact created something that can create numerous test cases - you just >>> > have >>> > to tell your PBT tool how many test cases you want to check the >>> > property on. >>> > >>> > Raising the bar by shrinking >>> > >>> > At this point you might still have the feeling that introducing the >>> > notion >>> > of some sort of generators to your unit testing tool of choice would >>> > bring >>> > you on par with PBT tools, but wait there is more to come. >>> > >>> > When a PBT tool creates a test case that fails there is real chance >>> > that it >>> > has created a long test case or some big input parameters - trying to >>> > debug >>> > that is very much like receiving a humongous log from a system in the >>> > field >>> > and try to figure out what cause the system to fail. >>> > >>> > Enter shrinking... >>> > >>> > When a test case fails the PBT tool will try to shrink the failing test >>> > case >>> > down to the essentials by stripping out input elements or events that >>> > does >>> > not cause the failure. In most cases this results in a very short >>> > counterexample that clearly states which events and inputs are required >>> > to >>> > break a property. >>> > >>> > As we go through some concrete examples later the effects of shrinking >>> > will >>> > be shown. >>> > >>> > ---------------------------------------------------------------------------------------------------------------------------- >>> > >>> > >>> > >>> > >>> > >>> > >>> > -- >>> > http://www.linkedin.com/in/torbenhoffmann >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "erlware-dev" group. >>> > To post to this group, send email to [email protected]. >>> > To unsubscribe from this group, send email to >>> > [email protected]. >>> > For more options, visit this group at >>> > http://groups.google.com/group/erlware-dev?hl=en. >>> > >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "erlware-dev" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/erlware-dev?hl=en. >>> >> > > > > -- > http://www.linkedin.com/in/torbenhoffmann > > -- > You received this message because you are subscribed to the Google Groups > "erlware-dev" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/erlware-dev?hl=en. > -- Martin Logan Erlang & OTP in Action (Manning) http://manning.com/logan http://twitter.com/martinjlogan http://erlware.org -- You received this message because you are subscribed to the Google Groups "erlware-dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/erlware-dev?hl=en.
