Some of the bugs it has helped me find that I am most appreciative of are in testing spatial code. One comes to mind when I developed the IsWithin predicate, and others in computing the bounding lat-lon box of a geodetic circle, and… and on and on… but the details are unimportant really and to tell you/your-audience would be a distraction. The point is, I don’t know where the bugs are a-priori; otherwise I’d fix them :-). When developing these tests, it’s almost pointless to put in particular static/fixed/boring test of the form: given this input, expect this output, unless I’m adding a regression test for a bug I found. Instead, the most thorough way to do it is to randomly come up with the shapes and have a brute-force way to test the result in the test, then test that the result is consistent when using the real code I want to test.
It’s not all roses… developing these tests are hard! — particularly when there is ambiguity in the answer (i.e. values being close-enough for the precision supported in the particular scenario). And diagnosing a failure takes time. And there’s a bit of an art to choosing random but not too random values. For example, have coincident & touching shapes, instead of purely random numerical values. Similar in concept is some random string generation utility methods I think you developed. ~ David Smiley Freelance Apache Lucene/Solr Search Consultant/Developer http://www.linkedin.com/in/davidwsmiley On Fri, Jan 2, 2015 at 4:00 AM, Dawid Weiss <[email protected]> wrote: > Hi everyone! > > I am giving a talk about randomized testing in a month or so and I > thought it'd be cool to share some real-life "ah-crap", "gotcha", > "wtf" moments related to the bugs discovered by randomized testing. > > My personal favorite is LUCENE-5168 (g1gc/ impossible code path) > which, by the way, is still unresolved and we have no idea how to > reproduce it reliably. Another is one of Robert's bugs with Unicode > where he debugged what seemed like a "megabutt" of crazy-hairy random > unicode that triggered the problem. These sorts of things. > > You can reply to the list or to my prv. e-mail if you wish. > > Dawid > > P.S. The talk's abstract is below here. > > Randomized Testing: When a Monkey Can do Better than Human. > > The talk will take a look at the concept of writing (unit) tests > with predictable randomness. We will try to generalize: > describe what such tests are, how to write them and how > randomized testing can help in finding bugs in otherwise > deterministic routines. We will also try to demonstrate on real-life > examples that human understanding and predictions with > regard to software are typically, ehm, incorrect (the same applies > to mice and towels, see: THGTTG for full explanation). > > Randomized testing has been used extensively in Apache Lucene, > Solr and ElasticSearch. Developers generally love it because it's effective > at finding bugs and hate it when the bugs it finds are theirs to debug. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
