On Tue, Nov 5, 2013 at 11:53 AM, Kathy Lussier <[email protected]> wrote:
I'm guessing this message was sent mostly to get developer feedback on
including unit tests. However, since there has been no developer feedback
and since this e-mail raised a lot of questions for me, I'll jump in with
those questions.
Kathy, I took Dan as trying to rally developers to actually move
forward with more testing.
<snip: automated testing>
it may need to be something that we might want to consider including in
future development contracts.
I think that's a grand idea. Equinox promises to do this with all of
its new development projects.
Can somebody provide an end-user explanation (please don't be afraid to dumb
it down as much as possible) as to how pgTAP tests help with the QA process?
In looking at the QA report, I see:
The particular part you quoted is about a test generator for creating
a baseline set of tests. This hasn't really been discussed widely,
and may or may not be a good idea. It's orthogonal to the idea of
using pgTAP in general. pgTAP tests will help just like other tests,
which I think you have a handle on below. The distinction between
unit and integration tests isn't that important; what's important is
coverage and quality of the tests, and the actual process of writing
the tests. Automated tests help us catch undesired changes in
software behavior. And the process of actually writing tests can
change how development is done, improving the development process by
changing how developers think about their work, even if a specific
test never actually catches a bug.
In looking at the QA report, I think I have a better handle on how
integration tests work. Basically, it looks like we have some scripts in
Evergreen that performs various functions, and the idea is to run those
tests when new code is added to make sure we continue to get expected
results. Is that assessment correct? The recommendation to continue to add
integration tests, then, is to make sure that any new functionality be
included as part of that testing to ensure future code changes don't break
that functionality, right?
Yes and yes.
I'm curious about the time/effort required if the recommendations from the
QA report were adopted. For those who have created pgTAP or integration
tests, how much of a time commitment do you think would be required to come
up to speed on creating these tests? Once developers are up to speed on
adding these tests, do you have any sense of how much time it would add to
the development process to include them?
For my part, it was mostly a mental barrier. Developers write code,
and tests are simply more code, and not a whole lot in comparison to
what is being tested.
A while back, there was some talk on the list about using Cucumber for
testing. http://markmail.org/message/otvljkdd4pwtg2ov My understanding from
the discussion thread was that it would make it easier for non-developers to
add tests. Is that something that could be used to help the community ease
into qa practices?
I think the concepts there could be very useful. I want to look at this more.
Thanks Kathy,
