Travis Vitek wrote:
[...]
I guess I hadn't thought of the fact that some of these tests
create multiple thread pools. Hmm. I don't think the test should
be allowed to reset its own timeout but I'm not sure I see a good
solution.
Yeah, ugh.
It certainly shouldn't exit successfully without doing
the rest of the work.
Agreed.
Should it fail?
Well, if we make the test fail aren't we essentially setting another
hard limit?
You're right, we would be for all the tests that create multiple
thread pools.
Or should it just continue with the next pool and allow
itself to be killed?
If we do that, then this timeout idea is done.
If the goal is to get the tests to run to completion sometime before the
hard limit there are a few obvious changes we could make. In some cases
we might be able to just reduce the default number of iterations.
The problem is that we don't know apriori how many iterations
each test will be able to complete in the specified amount of
[wall clock] time because the number depends on the hardware,
and on the system load at the time the test runs, in addition
to the test itself.
In
other cases, we could combine 'sections' of the test. Some tests have a
section to test a char specialization, then a wchar_t specialization
section, and finally a combined section that tests both. We could
potentially remove the first two sections, or at least disable them by
default.
Yes, that's a possibility.
Another option would be to continue testing these sections, but
separate them out into their own tests.
Right. That's the solution I keep ending up with as the most robust
one. The main advantage I see is minimizing the scope of the tested
functionality. The biggest disadvantage is the significant overhead
of building and running twice as many (thread safety) tests as we
have now. We could eliminate some of the overhead due to building
these tests by building just one binary and rigging our test driver
to invoke it two or more times with different options (I like this
feature in and of itself a lot) but I fear there is little we can
do to deal with the overhead of running the test(s) multiple times.
Over to you.
Martin
