Top-posting because my response (or, more appropriately, follow-up) is in the form of a public google doc.
Short version: We agree on all the goals, so I propose a way forward that leverages the size (bigger than pre-2.0 and growing) of our community and IMO gives us the tools to enact change toward those shared goals. Long version: https://docs.google.com/document/d/1EyLZ9PH25kwQvbC9uvYo0z8cYG5Ui3bFrzlkp2C2FYA/edit?hl=en_US Thanks, Dan, for clearly articulating your thoughts. I hope the doc above does the same. All, please, PLEASE take a few minutes to read that. Feedback strongly encouraged, both general and specific. If this thread grows a lot I'll happily open that doc for editing. --miker On Mon, Jun 6, 2011 at 1:34 AM, Dan Scott <d...@coffeecode.net> wrote: > On Sun, Jun 5, 2011 at 11:17 AM, Mike Rylander <mrylan...@gmail.com> wrote: >> On Sun, Jun 5, 2011 at 1:04 AM, Dan Scott <d...@coffeecode.net> wrote: > >>> * Are we ready to start making use of pgTAP? Changing the database >>> schema seems like a perfect use case for unit tests, to ensure that >>> expected behaviour is maintained through the upgrade, and to >>> demonstrate that buggy behaviour is fixed or non-existent behaviour >>> comes into existence via the upgrade. >> >> Ready? Sure. Tuit-ful? Not I... > > I'm not sure how to respond to this tactfully, so I won't try to be > clever or cute, I'll just be blunt. The alternative to putting in time > upfront on quality is to spend more time addressing quality problems > later after a release, and we've done a lot of the latter. We've had > trouble publishing high quality initial releases. Production sites > have been finding too many problems with their patrons and staff, and > it's not good for the Evergreen name. My hands are far from clean on > this front (hello, sites who upgraded from 1.6 -> 2.0 and ran into > problems with authorities), which is one of the reasons that I have > invested much of my own time in getting the continuous integration > server running again and creating a skeleton set of unit tests (and > thanks to Kevin for his efforts in that area too). It's also why I've > been a proponent of getting sign-off on branches from another > contributor instead of committing your own work directly to a core > branch. > > I believe that we can begin to address some of these quality issues > via more unit test coverage. I don't think that we're going to get > very far, though, if we just have one or two people trying to add unit > tests to other people's work - and those people are likely to have > their own areas of new functionality that they want to contribute to > Evergreen, rather than spending all of their time writing tests for > other people's code. The people creating new functionality or > modifying existing functionality are the ones who are in the best > position of knowing what inputs and outputs to expect from a given > chunk of code, and therefore to create basic unit tests demonstrating > those expectations - which helps other contributors weeks, months, or > years later know whether their own changes will break expectations. > But we need to adopt the approach as a team, not as individuals. > Tackling the database schema via pgTAP as modifications happen seems > like a small, reasonable step to take in this direction. It's not > trying to boil the ocean by saying that we need unit tests for every > function and every table in the database immediately; it's suggesting > that, when you modify the schema, you commit tests at the same time > that demonstrate that your changes do what you say they do (and > maintain existing behaviour). And eventually, I bet we would get a lot > of the database schema covered with this gradual approach. > > Unit tests alone won't prevent all of the problems that we've run into > with new releases, of course. I've been guilty of introducing new > functionality that proved to perform poorly at scale until indexes > were added, or that only showed up when data was migrated from a > previous release rather than loaded directly into the new release. > Bug #788379 ("broad searches are slow") is an example of a serious > performance regression in 2.0 that has yet to be addressed. > constrictor gives us some great tools on the performance testing > front, but it takes time to set up a clean environment loaded up with > sufficient data to trigger noticeable performance problems (let alone > tracking performance over time) or to run that environment through an > upgrade process and put the resulting environment through its paces. > We need repeatable upgrade tests and performance tests - maybe a > community environment that runs a standard set of system tests on a > regular basis and tracks those results over time? > > In summary, I don't think I'm the only person who feels that we've had > quality problems. There are probably ways to address these problems > that I haven't raised here, and I'd be happy to hear about > alternatives from people who are prepared to adopt them. I just don't > want to see a 2.1.0 release that isn't really ready for prime time > until 2.1.6, and a 3.0 release that isn't ready for adoption until > 3.0.6, and I don't want libraries playing a game of chicken to see who > is willing to be the early adopter of a new release. I want libraries > confident that they can adopt a *.*.0 release, and I want them to be > proud of Evergreen's quality and able to recommend it without > reservations to other libraries. > -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / Your Library's Guide to Open Source | phone: 1-877-OPEN-ILS (673-6457) | email: mi...@esilibrary.com | web: http://www.esilibrary.com