Travis Vitek wrote:
Martin Sebor wrote:
[...]
We could eliminate some of the overhead due to building
these tests by building just one binary and rigging our test driver
to invoke it two or more times with different options (I like this
feature in and of itself a lot) but I fear there is little we can
do to deal with the overhead of running the test(s) multiple times.
Are you talking about modifying the nightly build infrastructure so that
it will invoke a test several times with different command line
arguments to run the test differently, or are you thinking about having
the test invoke itself with different arguments or something? I don't
think any option other than the first will work.
The former. I'm envisioning changing the exec utility (which runs
the tests) to look for some "command" file corresponding to each
test from which it would figure out how many times to run it and
with what options. If the file didn't exist exec would run the
test the same way it does now.
This approach could even be extended to compiling and linking
certain tests with special options, or to implement negative
testing, i.e., to exercise the ability of the library to reject
invalid programs. It would even let us deal with the increasing
disk space problem by compiling and linking each test on demand
just before running it and then immediately deleting it to free
up disk space.
What do you think about this?
Martin
