Eric Lemings wrote:
-----Original Message-----
From: Martin Sebor [mailto:[EMAIL PROTECTED]
Sent: Friday, April 25, 2008 1:03 PM
To: [email protected]
Subject: Re: Regression tests file structure
Eric Lemings wrote:
Not that it really matters but I noticed lots of rather
small programs
for the regression tests. Couldn't these be consolidated into fewer
source files, by component perhaps, with a function for
each individual
regression test? Just a thought.
I'm sure they could, and it would probably benefit our build
times. The problem is that a severe failure in one of them
(build or runtime) would cause the rest of them to fail too.
Getting around it would require non-trivial changes to our
build harness (e.g., build each test into a separate object
file, linking them all together into a single executable,
and on a runtime failure in one re-link all those that
didn't get to run into a new executable, and repeat).
Mmm, that's not quite what I had in mind. Here's an example of what I
had in mind:
test/regress/21.string.cpp:
...
static void
test_stdcxx_162 () {
...
}
static void
test_stdcxx_231 () {
...
}
static void
test_stdcxx_466 () {
...
}
static void
run_tests () {
test_stdcxx_162 ();
test_stdcxx_231 ();
test_stdcxx_466 ();
}
All regression tests related to the same component (i.e. clause) are
contained in the same source file. That wouldn't require build/test
harness changes, would it? And yes a build or run failure would cause
them all to fail but I'd see that as real incentive for correcting the
failure. :)
We certainly could structure the tests this way. It would alleviate
the fatal failure problem to some extent but it wouldn't completely
eliminate it.
I'm not sure that the incentive argument is any more powerful for
regression tests than it is for the others, and look at how many
failures we still have in the test suite. It would need to be
weighed against the benefits of this restructuring. What do you
see as the main benefits of this change?
Martin
