To start with, the idea of some kind of "switch" to turn all failures into TODOs has been removed. It was a bad idea from the start and since it never actually was implemented, I have ditched it entirely.
Second, I would like to move away from the force_todo() usage, as it has a number of obvious problems. However, I cannot dismiss it's usefulness as a quick-fix during release preparation, so i will not yet remove it.
It was determined (on #perl6 and various emails) that what we really need in Pugs is a multi-layered TODO model. In Pugs we have several different kinds of TODO tests, along with the plain old TODO. Here is a list of some of the more common ones and examples of how the attributes might look;
- TODO
:todo, :todo(1)
Sometimes, it just doesn't fit anywhere else, so in that case it is just TODO.
- TODO bug
:todo<bug>, todo => 'bug', :todo('bug')This is either an outright bug in Pugs, or it is a feature which /should/ be implemented, but for some reason (usually some kind of seemingly unrelated bug) it is not. This is a good "layer" for edge cases tests and other such ugly corners.
- TODO feature
:todo<feature>, todo => 'feature', :todo('feature')These are tests written for perl6 features which are not yet implemented in Pugs. Objects is the perfect example of this.
- TODO parsefail
:todo<parsefail>, todo => 'parsefail', :todo('parsefail')These are tests which fail to parse correctly. These are usually tested with eval_ok() or eval_is().
- TODO hardfail
:todo<hardfail>, todo => 'hardfail', :todo('hardfail')These are tests which parse okay, but die with an uncatchable error. I will be experimenting with using the "is lazy" trait to catch and not-evaluate these (and therefore not kill pugs).
My goal was to support these different kinds of TODOs with Test.pm, and also make them visible through Test::Harness (and eventually in our smoke test). In exploring Test::Harness I noticed that TODO tests can also have a "reason", just like SKIP tests do (although Test::Harness seems to ignore them) and I have exploited this feature to accomplish my goal.
So currently (in the latest revision), Test.pm will support the following code:
ok(bad_function(), "... testing bad_function", :todo<bug>);
this will produce the following TAP output:
not ok 1 - ... testing bad_function # TODO bug # Failed (TODO bug) test (-e line 3, column 1-46) # Got: undef
I have also altered our local copy of Test::Harness (in inc/) so that if you were to run this with 'prove' or 'make test' you would see something like this:
t/my_test_file....ok
1/5 TODO bug test
2/5 TODO tests
2/5 TODO feature tests
All tests successful.This allows a simple top-level overview of not only the number of TODO tests, but of each specific TODO test "layer". Ideally we can support this level of TODO granularity in the test smoke interface as well.
Now all that is left is for people to start changing all the :todo tests to :todo<some_attribute>.
As always comments, question and suggestions are always welcome.
Thanks,
Stevan
