I am a casual developer in the Evergreen community. Here are comments of mine about some excerpts from Sustainable QA that I find particularly interesting. The comments are primarily technical in nature but I think of general interest.
p.4, 'Providing example tests at each major level of the Evergreen “stack” and making it straightforward to run such tests whenever a copy of the source code is run through the build procedure will significantly facilitate the writing of additional unit tests by any party.' As a casual contributor to the code, I look forward to having the above state of affairs. It's been about seven years since the first releases of the software, and I do not understand why the testing framework is not more mature. Whenever I contribute a bit of new logic to the Perl code, for example, I am stymied by not being able to contribute a unit test to show evidence to others that the new logic works. p.4, 'Today Evergreen has a handful of unit tests in Perl.' p.1, 'Some components of QA exist today in Evergreen and OpenSRF. We have Perl and C unit test frameworks. We have a load test driver in Constrictor. We have continuous integration infrastructure using Buildbot to run basic code verification.' The Buildbot[1] is currently not very capable and therefore not very significant in its impact. Of the 315 tests that the Buildbot reports as succeeding, most of them check if Perl modules are installed, but do not execute any source code. Of the few unit tests that actually runs, none of them can test an OpenSRF service call nor perform a database operation. This is because the 'continuous integration infrastructure' takes effect during 'installation time' and does not presume the OpenSRF framework or the Postgres database is installed and running. p.4, 'To effectively test the Evergreen server as a whole it is necessary to issue API requests against a running server with sample data. Such requests exercise each layer of Evergreen, including the database, the backend modules and the code that allows them to communicate' To achieve the above, the Buildbot must have OpenSRF (and ejabberd, the XMPP server) installed. If there was a way to 'stub out' OpenSRF, many functional tests could be run earlier in the process, ie, without setting up a fully running server. By 'stubbing out' OpenSRF, I mean to have a situation where the OpenILS code runs with OpenSRF replaced by something simpler[2]. [1] http://testing.evergreen-ils.org/buildbot/waterfall [2] I have been investigating how OpenSRF can be stubbed out; it looks like some systemic code changes are needed in the OpenILS application modules in order for a simple stub to work. -- steven chan [email protected]
