--- Tony Bowden <[EMAIL PROTECTED]> wrote: > We insist on a full regression test on checkin, so having the test suite > take a long time to run puts people off making small changes. :( > > On one project a full regression test takes about 8 minutes, which is at > least an order of magnitude longer than I'd like. I couldn't dream of > having it take an hour!
Unfortunately, that's something we seem to be stuck with here. At my previous job, we had one
test suit with a similar number of tests, but it took -- at most -- about 3 minutes to run. The
problem is that test driven development started here before the current testing craze. As a
result, my fellow programmers have some curious notions about testing. Amongst other things, each
file is a package that has several TEST subroutines in it. These tend to have the following
characteristics:
sub TEST__WE_CAN_ADD_FOO_TO_BAR { make_up_test_data(); add_test_data_to_database(); my $foo = function_to_test(); assert($foo, $some_made_up_val); }
Now, before that function is run, we automatically have a "setup_test" function that runs. After
that function runs, a "teardown_test" function runs (which typically does a rollback).
So we have to make up a bunch of dummy data, insert this data, test this data, and roll back this
data FOR EVERY TEST. Further, because of cut-n-paste testing, much of the dummy data that is
inserted has nothing to do with a particular test and therefore is useless overhead. Not only
does this slow the test suite to a crawl (and this is with Oracle, no less), I've asked some of my
fellow programmers if they see a problem with testing made-up data and all I get are blank stares.
It's similar to why programmers should not do their own acceptance testing: two minutes later
they send an email saying "yup, it does what I programmed it to do."
To make matters worse, I've also asked "have you ever considered testing the data that you have in
production?"
The answer, from every programmer I've talked to, has been "how can we test the data if we don't
know what it is?" The programmes working here haven't worked with any other way of testing so it
never occurs to them that there might be another way.
This company is doing some great work, but sometimes I want to scream.
Cheers, Ovid
There is another side to the "made-up" data issue. There are instances where you legally can _not_ use live data. Consider a Hospital or Insurance company, for example ...
What we did was take a proper subset of the live data and sanitize it to remove identifiable things (name, SSN, Medical Number, address, you get the idea). It took a lot of work to get right; but it did give us a feel for what real data looked like. We rebuilt the test DB every three months, so trends in data quality could be factored in.
This meant that running tests did not have to do a setup/tear-down unless they were exercising Objects not yet in production. It also gave us heads up on code that was doing excessive searches or otherwise abusing the database and the User's patience.
B Bob Goolsby [EMAIL PROTECTED] (408) 853-2055
