I'm guessing this message was sent mostly to get developer feedback on
including unit tests. However, since there has been no developer
feedback and since this e-mail raised a lot of questions for me, I'll
jump in with those questions.
Overall, I support better QA practices and have been following much of
the QA discussion over the past few months. However, after reading this
e-mail, I realized I don't have a good understanding of how these tests
lead to better quality. Since MassLNC sponsors a lot of development
projects, I think it's important that I have a better understanding of
these tests since it may need to be something that we might want to
consider including in future development contracts.
Can somebody provide an end-user explanation (please don't be afraid to
dumb it down as much as possible) as to how pgTAP tests help with the QA
process? In looking at the QA report, I see:
The idea here is that if we can get a commitment from the core
developers, we can use this script once to create a base set of tests
that we can then commit to the master branch. From then on we can
simply maintain and modify those tests whenever we create a database
upgrade script. These tests can then prove that both the upgrade
scripts and the seed files do the same thing, and folks performing
upgrades can sanity-check their work to ensure that they didn't miss
invoking an upgrade script, or that an upgrade script somehow failed
unnoticed.
Is the intent of the test to verify that that database changes make it
into the upgrade scripts? Is it also doing some kind of check to make
sure the database change doesn't break anything? Is the test only useful
for that one database change or is it something that is then part of a
suite of tests that are run whenever new database changes are added?
In looking at the QA report, I think I have a better handle on how
integration tests work. Basically, it looks like we have some scripts in
Evergreen that performs various functions, and the idea is to run those
tests when new code is added to make sure we continue to get expected
results. Is that assessment correct? The recommendation to continue to
add integration tests, then, is to make sure that any new functionality
be included as part of that testing to ensure future code changes don't
break that functionality, right?
I'm curious about the time/effort required if the recommendations from
the QA report were adopted. For those who have created pgTAP or
integration tests, how much of a time commitment do you think would be
required to come up to speed on creating these tests? Once developers
are up to speed on adding these tests, do you have any sense of how much
time it would add to the development process to include them?
A while back, there was some talk on the list about using Cucumber for
testing. http://markmail.org/message/otvljkdd4pwtg2ov My understanding
from the discussion thread was that it would make it easier for
non-developers to add tests. Is that something that could be used to
help the community ease into qa practices?
Thank you all for your patience as I try to gain a better understanding
of these issues.
Kathy
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 343-0128
[email protected]
Twitter: http://www.twitter.com/kmlussier
On 11/1/2013 5:01 PM, Dan Scott wrote:
In the QA report for which several members of the community generously
paid $30,000, the "Moving Forward" section at
http://nox.esilibrary.com/~jason/qareport/qa.html#_moving_forward
states:
"We recommend that the development community start including integration
tests with their changes to the backend, and pgTAP tests with their
database changes (there was discussion and general interest in this
during a developers meeting)."
The referenced developer meeting is minuted at
http://evergreen-ils.org/meetings/evergreen/2013/evergreen.2013-08-27-14.04.html
with the agreement "general interest in easing into qa practices
demonstrated by phasefx, phasefx to hold hands with everyone, especially
through the mailing list"
I'm concerned that since then we have seen a number of database changes
without corresponding pgTAP tests (the most recent pgTAP test was
committed Sept 3rd, while there were a ton of changes committed through
October). I did bring this up in the IRC channel with respect to one of
the recent bug fixes that was committed for a fairly fundamental
function, but my gentle prod in that case appears to have been
overlooked.
I recognize there is pressure to get the 2.5 release out, but if we
continue to follow our past approach of not including unit tests where
the path has been blazed for us when we commit changes to the database,
our quality assurance is going to assuredly be similar to the quality we
have produced in the past.
In short, we have heard strongly from the community that we are not
producing software of the quality that they expect; and the community
has followed up those words with a significant investment in the QA
project; and we as a developer team are _not_ following through with the
process improvements we had agreed to adopt.
FWIW, should anyone want to follow some commits to teach themselves how
to add pgTap tests, I did go through the learning process to support
https://bugs.launchpad.net/evergreen/+bug/1242999 - the tests I created
and the instructions for running them are basic, but they're a start.
Thanks to Galen for giving me a pointer in the right direction.
