On Tue, Feb 19, 2013 at 03:49:26PM -0800, James Fournie wrote: > Hi there, > > One of the suggestions I made with regard to QA in the recent IRC > meeting, was to look into Cucumber for testing. I thought I'd follow > up a bit. Caveat that this is not a QA silver bullet, just exploring > some thoughts on it and how it could be used. > > Cucumber is a Ruby-based tool for testing, based around > behaviour-driven-development principles. A key part of Cucumber, is > the Gherkin language is uses. Gherkin is a business-facing language, > meaning it's designed to be understandable by non-developers.
<snip> > As you can see, it's pretty simple to understand. My colleague Steven > Chan in a Sitka team discussion noted that we do a lot of general > testing in-house with each upgrade and it would be nice to capture > that somehow. I am sure that other Evergreen implementors have this > knowledge drain as well and I think that Gherkin might be a way to > capture some of that. > > I've put together a basic Cucumber Ruby project for testing the TPAC > mainly as a proof of concept. Look in the 'features' dir at the > 'feature' files for the Gherkin: > > https://github.com/jamesrf/evergreen-ils-acceptance-test This looks _awesome_, James. I've heard about Cucumber before but I think my Perl/Python oriented brain basically ruled it out due to the Ruby bit. But... that shouldn't matter for the purpose of testing. And there is a _ton_ of what you've said that overlaps with the discussion that was held during the hack-a-way. We had been struggling, a bit, with how to capture the repeatable testing workflows and thinking about the wiki as a repo, to start with, but there would be so much value to the Cucumber approach. An immediate win would be to extend your search examples to build on top of the sample data we already have available to cover the kind of searches that Thomas refactored with the QueryParser fix branch--to ensure that we continue to get the results that we always wanted from the previously working searches, and to ensure that we start getting the results that we always wanted from the broken searches. We need to move this beyond a proof of concept, this is a great concrete way of moving the quality of Evergreen forward. Thank you for taking things this far so far!
