This is cool. I'd say lets get it up and going on jenkins (even weekly or something). why worry about the imperfections in any of these coverage tools, whats way more important is when the results find situations where you thought you were testing something, but really arent, etc (here was a recent one found by clover http://svn.apache.org/viewvc?rev=1376722&view=rev).
so imo just another tool to be able to identify serious gaps/test-bugs after things are up and running. and especially looking at deltas from line coverage to identify stuff thats 'executing' but not actually being tested. some of that is random tests, but even then its still useful. its not great if the tests are inefficient either (take 20 runs to find a bug), even though its good that hopefully they will eventually. so we can identify situations like that and make it easier to work on the codebase (e.g. merge a feature branch without running tests on it 1000 times to be confident first) On Mon, Aug 27, 2012 at 7:16 PM, Greg Bowyer <[email protected]> wrote: > Hi all > > At my current $DAYJOB we have been having a bit of success with an > alternative coverage tool called pit-test ( http://pitest.org/). Essentially > pit-test is a mutation testing tool that attempts to see how well the unit > tests are able to catch alterations and regressions in the code that they > aim to test; this is done by determining what code the test actually touch > and then mutating that code in some fashion. > > I have as such started working on seeing if I can integrate pit-test into > the lucene build, the tool itself is apache licensed which solves that > particular issue. > > The main downsides I can see is that the coverage might be different across > runs due to lucenes random testing, as well as the time it takes to run > coverage (~4 hours on a big bad sandy bridge machine). > > I have published initial results for lucene packages, core is missing > because there is one test in core that does not work with pit-test currently > (and as such I am currently working on getting core generated. > > It can be found here http://people.apache.org/~gbowyer/pitest/ > > Is this of interest to anyone other than myself, especially given the > aggressive nature of how lucene is tested ? > > -- Greg -- lucidworks.com --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
