On May 16, 2010, at 8:13 PM, David Chelimsky wrote: > On May 16, 2010, at 12:54 PM, Scott Taylor wrote: > >> Hey all, >> >> I'm wondering if anyone has any experience with an automated test-case >> generation tool like Quickcheck (for erlang/haskell). I'd be interested in >> hearing any impressions, war stories, or dev workflows regarding a tool like >> this. Talking off list to David C, he suggested that it might be a >> complimentary tool to a TDD/BDD framework like rspec. > > My thinking here is that it could be useful to drive out an initial > implementation using TDD, and at the point we think we've got the solution we > want, add something quickcheck-like to try to poke holes in it. I'd probably > then add new examples if any cases I hadn't considered were revealed through > this process.
Have you watched John Hughes' presentation on the matter? http://video.google.com/videoplay?docid=4655369445141008672# It's sort of interesting that he won't do any TDD - he'll let the reduction process generate the "minimum" test case, and go from there (that's not explicitly stated in that video, although I'm pretty sure I've heard him say it before). If I had a tool like this, I'm guessing I'd probably have a workflow like the following: 1. use the random test case generator, and fix any issues that were obvious. 2. If something wasn't obvious, I'd go and write a test case for in a more traditional testing tool (rspec). I often use the debugger in conjunction with the spec runner, running the one test case with a debugger statement at the start of the test case. 3. Any regressions would (obviously) happen in the traditional tool. The big win with a tool like this is not testing boundary cases, it's in having the tool "write" the test cases for you. OTOH, I wonder if the simplicity of the implementation would be sacrificed when taking this approach. Another drawback - I have no idea how such a tool would integrate with a build server. > >> It appears as though there is a similar project out there for ruby named >> rushcheck (http://rushcheck.rubyforge.org/). > > It's up on github too: http://github.com/hayeah/rushcheck. Same guy has this > too: http://github.com/hayeah/rantly - random data generator - looks like you > could do stuff like: > > Rantly.new.each(100) do > thing.method_that_accepts_a_string(string).should have_some_quality > end There's a blog post about the library here, if anyone is interested: http://www.metacircus.com/hacking/2009/04/10/look-ma-no-monads.html I've been thinking about integrating a port of the ruby library faker into scriptcheck, the javascript testing tool I've been working on: http://github.com/Marak/Faker.js http://github.com/smtlaissezfaire/scriptcheck > > This would cause 100 random strings to be generated and passed to > thing.method_that_accepts_a_string. Assuming the matcher verifies some set of > rules about the outcomes, you've basically got quick check. Yeah, pretty much. One issue, though, is that you don't want to hard code the number of random generations. You'll also want a convenient way to run just one given test case easily (which rspec already has). You'll probably also want to separate these random generation tests from the rest of your tests. Hitting a database 1000 times for one test is going to be costly. Now that I'm thinking about it, it might make a ton of sense in languages like erlang or haskell where everything is functional because those languages lend themselves to parallelization since there are no shared resources. Regards, Scott _______________________________________________ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users
