On Fri, Mar 25, 2011 at 00:34, Jordan Wilberding <[email protected]>wrote:
> I am just thinking surely you could calculate the probability that a > certain test case(or error producing state) will be repeated over time given > certain constraints. Maybe not though. I am just thinking out loud mostly at > the moment. I still need to get a better grasp of property based testing > before I formulate a potential way for avoiding the problem Eric was talking > about with respect to the need to make sure a previously found error > producing state is accounted for in the future. > Now I can see what you are looking for. When a test case has failed it would be nice to know the probability of an isomorph test case being generated again - this is dependent on: the number of tests to be generated and the nature of the generators. Increasing the number of test will raise the probability - no surprises there. On the generator side things start to become more muddy. The integer() generator will pick a random integer inside the space of what integers can be in Erlang, but it will do so with small numbers first and then grow them. This makes the probability calculations rather hairy. One can generate statistics about the range of test cases ones generators create and that is something I will cover in the "PBT for UT" document - it is not the first thing to throw at people! The only way to be 100% sure about accounting for an error producing test case going forward is to store the test case and make it part of a statical set of tests that one executes on top of the generated test cases. As Anders stated earlier you might want to explore an area where you have found a problem by improving on your generators, but it will not give you 100%. For the big system test thing I presented at EUC 2010 we never stored the failing test cases. On of the reasons was that it was very likely that the same test case would be part of the generated set in the future, so the burden of storing and executing static test outweighed the benefits. It depends on the situation, so I suggest that we ensure that this aspect is covered in the tutorial and then we get some experiences as we spread PBT in the Erlware code base. Cheers, Torben > > Thanks! > JW > > On Thu, Mar 24, 2011 at 4:28 PM, Torben Hoffmann > <[email protected]>wrote: > >> On Thu, Mar 24, 2011 at 17:12, Jordan Wilberding >> <[email protected]>wrote: >> >>> Can we use a monte carlo model, where we can guarantee things to be >>> failure-free with a certain amount of probability? >>> >>> >> I don't think so, but I am not an expert on Monte Carlo models! >> >> Could you explain how a MC model can be used to predict failure-free >> probabilities? >> Normally you would use MC simulation to get a prediction of a range, e.g., >> when you are modelling when a series of tasks with given distribution will >> be done if executed right after one another. >> >> What you can do with most PBT tools is to collect code coverage numbers >> while you test as well as statistics on what sort of tests has been >> generated. >> Is that in the direction you were thinking of? >> >> Cheers, >> Torben >> >> >> >>> Thanks! >>> Jordan Wilberding >>> >>> >>> On Thu, Mar 24, 2011 at 10:10 AM, Eric Merritt >>> <[email protected]>wrote: >>> >>>> Torben, >>>> >>>> Great first stab I think, and I suspect its going to turn into a good >>>> proper article. I am really curious about narrowing, or specifically >>>> reproducible failures. >>>> >>>> The impression I get is that the generated values are generated, that >>>> is there is some level of randomness in the generation. If that is the >>>> case, how do you make property test failures reproducible? I also >>>> suspect this is where narrowing comes in to play. >>>> >>>> Eric >>>> >>>> On Wed, Mar 23, 2011 at 5:14 PM, Torben Hoffmann >>>> <[email protected]> wrote: >>>> > Hi, >>>> > >>>> > I have started on my explantion of what property based testing is - it >>>> is >>>> > not done yet, but I thought I would release early and get some >>>> feed-back so >>>> > I can avoid having text that requires too much inside knowledge. >>>> > >>>> > Please comment and provide alternative descriptions - the goal is to >>>> get a >>>> > text that will help people that wants to get into property based >>>> testing and >>>> > once you have learned it it will be so much more difficult for you to >>>> spot >>>> > insider information. >>>> > >>>> > Cheers, >>>> > Torben >>>> > >>>> > >>>> ---------------------------------------------------------------------------------------------------------------------------- >>>> > Property based testing for unit testers >>>> > >>>> > The purpose of the short document is to help people who are familiar >>>> with >>>> > unit testing understand how property based testing (PBT) differs, but >>>> also >>>> > where the thinking is the same. >>>> > >>>> > This document focusses on the PBT tool PropEr for Erlang since that is >>>> what >>>> > I am familiar with, but the general principles applies to all PBT >>>> tools >>>> > regardless of which language they are written in. >>>> > >>>> > The approach taken here is that we hear from people who are used to >>>> working >>>> > with unit testing regarding how they think when designing their tests >>>> and >>>> > how a concrete test might look. >>>> > >>>> > These descriptions are then "converted" into the way it works with >>>> PBT, with >>>> > a clear focus on what stays the same and what is different. >>>> > >>>> > Testing philosophies >>>> > >>>> > A quote from Martin Logan: >>>> > >>>> > For me unit testing is about contracts. I think about the same things >>>> > I think about when I write statements like {ok, Resp} = >>>> > Mod:Func(Args). Unit testing and writing specs are very close for me. >>>> > Hypothetically speaking lets say a function should return return {ok, >>>> > string()} | {error, term()} for all given input parameters then my >>>> > unit tests should be able to show that for a representative set of >>>> > input parameters that those contracts are honored. The art comes in >>>> > thinking about what that set is. >>>> > >>>> > The trap in writing all your own tests can often be that we think >>>> > about the set in terms of what we coded for and not what may indeed be >>>> > asked of our function. As the code is tried in further exploratory >>>> testing >>>> > and >>>> > in production new input parameter sets for which the given function >>>> does not >>>> > meet >>>> > the stated contract are discovered and added to the test case once a >>>> > fix has been put into place. >>>> > This is a very good description of what the ground rules for unit >>>> testing >>>> > are: >>>> > >>>> > Checking that contracts are obeyed. >>>> > Creating a representative set of input parameters. >>>> > >>>> > The former is very much part of PBT - each property you write will >>>> check a >>>> > contract, so that thinking is the same. Note: the word property is >>>> used >>>> > instead of test case since you the property says something about your >>>> system >>>> > in general and not just a single run of things as with a test case. >>>> More on >>>> > this right away! >>>> > >>>> > What PBT does different is the input parameters. Instead of crafting a >>>> few >>>> > off-the-top-of-your head input parameters you specify the rules for >>>> creating >>>> > the input parameters and let your property based testing tool generate >>>> the >>>> > input parameters. >>>> > >>>> > The functions you write to specify the rules are called generators. >>>> > >>>> > Suppose you want to test a sorting function that takes a list of >>>> integers >>>> > and returns the list sorted. >>>> > The generator for this would be a function that returns a list with a >>>> random >>>> > number of elements, each of which is a random number. >>>> > >>>> > One can argue that that sort of randomness can be applied to unit >>>> testing as >>>> > well and indeed it can. It is just easier with a good PBT tool since >>>> it has >>>> > facilities that allows you to state the above generator as: >>>> > list(integer()) >>>> > >>>> > But it does not stop at generation of input parameters. If you have >>>> more >>>> > complex tests where you have to generate a series of events and keep >>>> track >>>> > of some state then your PBT tool will generate random sequences of >>>> events >>>> > which corresponds to legal sequences of events and test that your >>>> system >>>> > behaves correctly for all sequences. >>>> > >>>> > So when you have written a property with associated generators you >>>> have in >>>> > fact created something that can create numerous test cases - you just >>>> have >>>> > to tell your PBT tool how many test cases you want to check the >>>> property on. >>>> > >>>> > Raising the bar by shrinking >>>> > >>>> > At this point you might still have the feeling that introducing the >>>> notion >>>> > of some sort of generators to your unit testing tool of choice would >>>> bring >>>> > you on par with PBT tools, but wait there is more to come. >>>> > >>>> > When a PBT tool creates a test case that fails there is real chance >>>> that it >>>> > has created a long test case or some big input parameters - trying to >>>> debug >>>> > that is very much like receiving a humongous log from a system in the >>>> field >>>> > and try to figure out what cause the system to fail. >>>> > >>>> > Enter shrinking... >>>> > >>>> > When a test case fails the PBT tool will try to shrink the failing >>>> test case >>>> > down to the essentials by stripping out input elements or events that >>>> does >>>> > not cause the failure. In most cases this results in a very short >>>> > counterexample that clearly states which events and inputs are >>>> required to >>>> > break a property. >>>> > >>>> > As we go through some concrete examples later the effects of shrinking >>>> will >>>> > be shown. >>>> > >>>> ---------------------------------------------------------------------------------------------------------------------------- >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > -- >>>> > http://www.linkedin.com/in/torbenhoffmann >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups >>>> > "erlware-dev" group. >>>> > To post to this group, send email to [email protected]. >>>> > To unsubscribe from this group, send email to >>>> > [email protected]. >>>> > For more options, visit this group at >>>> > http://groups.google.com/group/erlware-dev?hl=en. >>>> > >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "erlware-dev" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/erlware-dev?hl=en. >>>> >>>> >>> >> >> >> -- >> http://www.linkedin.com/in/torbenhoffmann >> > > -- http://www.linkedin.com/in/torbenhoffmann -- You received this message because you are subscribed to the Google Groups "erlware-dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/erlware-dev?hl=en.
