On Apr 26, 2008, at 2:42 PM, Martin Sebor wrote:
Eric Lemings wrote:
-----Original Message-----
From: Martin Sebor [mailto:[EMAIL PROTECTED] Sent: Friday,
April 25, 2008 1:03 PM
To: [email protected]
Subject: Re: Regression tests file structure
Eric Lemings wrote:
Not that it really matters but I noticed lots of rather
small programs
for the regression tests. Couldn't these be consolidated into
fewer
source files, by component perhaps, with a function for
each individual
regression test? Just a thought.
I'm sure they could, and it would probably benefit our build
times. The problem is that a severe failure in one of them
(build or runtime) would cause the rest of them to fail too.
Getting around it would require non-trivial changes to our
build harness (e.g., build each test into a separate object
file, linking them all together into a single executable,
and on a runtime failure in one re-link all those that
didn't get to run into a new executable, and repeat).
Mmm, that's not quite what I had in mind. Here's an example of
what I
had in mind:
test/regress/21.string.cpp:
...
static void
test_stdcxx_162 () {
...
}
static void
test_stdcxx_231 () {
...
}
static void
test_stdcxx_466 () {
...
}
static void
run_tests () {
test_stdcxx_162 ();
test_stdcxx_231 ();
test_stdcxx_466 ();
}
All regression tests related to the same component (i.e. clause) are
contained in the same source file. That wouldn't require build/test
harness changes, would it? And yes a build or run failure would
cause
them all to fail but I'd see that as real incentive for correcting
the
failure. :)
We certainly could structure the tests this way. It would alleviate
the fatal failure problem to some extent but it wouldn't completely
eliminate it.
I'm not sure that the incentive argument is any more powerful for
regression tests than it is for the others, and look at how many
failures we still have in the test suite. It would need to be
weighed against the benefits of this restructuring. What do you
see as the main benefits of this change?
Better organization mostly. Fewer test programs with only one test
case. I figure if you're adding a new test program, it should have at
least a few test cases, not just one. If you're just adding just a
new test case, there's probably a related test program already in the
test suite.
No, these aren't huge benefits but neither are the costs. As I said
opening the thread, "not that it really matters..."
Brad.
