On Mon, May 12, 2008 at 08:41:58PM +0200, Bram wrote:
> I'm not saying that it shouldn't change.
> What I'm saying is that the value may change but that if it changes I
> would like to be informed about it.
>
>
> >Leave your todo test as it was to start with.
> >
> >Create a new test file "development_values_that_shoulnot_change.t" in your
> >developer test directory (that's not just for pod right).
>
> Which is not practical at all.
> That means maintaining an extra test file for each file that defines a
> todo test and that means duplicating every todo test.
My view too. It adds a lot of complexity. (but)
> What I'm suggesting is that it outputs something similar for TODO test
> that return an unexpected result.
>
> Again, make test only outputs All test succesful.
> Users will only run make test and thus can't be confused by the extra
> output.
I think that this problem could be solved without chances to TAP.
Specifically, one is trying to make a way of specifying two alternate
"expected" results for a given test.
One is the right one. If this result appears, then TODO test unexpectedly
succeeded. (Which is what we have currently)
One is the current wrong one. If this result appears, then the current test
failed, but it's TODO. (Which, at the TAP level, is what we have currently)
The third case is "anything else". Right now, it's also "not ok (TODO)" and
the TAP parser assumes that it's not noteworthy.
What might be needed is a coupled way of passing *two* tests into the test
running framework (Test::Builder or the like). One is the "desired" result.
One is the "current expected fail" result. Right now the only thing we test
is the "desired" result.
The framework then spits out the results from both tests (run the desired
result first) as
current expected result
ok not ok
desired ok "can't happen" "not ok TODO,
not ok TODO"
result not ok "not ok TODO, "not ok
ok TODO" not ok TODO"
"can't happen" is also reported as "not ok" - it's a critical logic error in
the tests.
(ie top left and bottom right are fails, bottom left and top right are the two
expected cases, and so both pass, although bottom left is "unexpected TODO
pass")
Could this be done with a Test::Builder (subclass) function that takes two
subroutine references - one that tests for the "desired" result (and returns
it as the return value), and one that tests for the "current expected result"
(and likewise returns it as the return value).
With suitable coding it would be possible in the actual test to call it quite
tersely, with minimal code duplication, possibly using closures.
One will still get all the diagnostics from running the test twice.
It would just be that the second test, the one we currently don't have a way
to write, would loose its "TODO" and become a hard fail if things change in
the wrong direction.
Nicholas Clark
