On Jan 23, 8:59 pm, jswit...@gmail.com (Jason Switzer) wrote:
> That sounds useful on the surface but often turns out to be more difficult
> to do than you might think. There are many cases where tests are performed
> from within loops. Something like S09.237 may or may not be in a loop, may
> be difficult to identify in files with many tests.
There are at least two reasons to identify a test (or check): to
control it from afar, and to track it's results.
If the reason for wanting identity is to control it (e.g.
Foo::Bar::Test<S09.237>.disable()), then the fact that it's in a loop
isn't necessarily important: if you want to disable it, then you
probably want to disable all iterations. If we do want finer grain
control, then it is probably possible to do something with resumable
exceptions that are thrown each time the test is potentially skipped.
If the reason to identifying a check is to track its result, then the
obvious solution is to not assume that it's result is pass/fail, but
is instead a pair of pass/fail counts (or pass/total -- same thing). A
good testing approach is "directed-random", where the same test is run
multiple times with different random seeds so as to use different test
data. IMO, it is reasonable to think of a one-shot test as an