Nathan,
On May 3, 2005, at 12:32 PM, Nathan Gray wrote:
On Tue, May 03, 2005 at 10:36:40AM -0400, Stevan Little wrote:The ongoing search to find a good model for TODO tests in Pugs continues ...
To start with, the idea of some kind of "switch" to turn all failures
into TODOs has been removed. It was a bad idea from the start and since
it never actually was implemented, I have ditched it entirely.
Good.
Yeah it was a cheap hack, so best left undone :)
Second, I would like to move away from the force_todo() usage, as it
has a number of obvious problems. However, I cannot dismiss it's
usefulness as a quick-fix during release preparation, so i will not yet
remove it.
That also sounds good. If not sure, wait a bit.
Agreed. Even cheap hacks can be useful :)
It was determined (on #perl6 and various emails) that what we really
need in Pugs is a multi-layered TODO model. In Pugs we have several
different kinds of TODO tests, along with the plain old TODO. Here is a
list of some of the more common ones and examples of how the attributes
might look;
This sounds like a really good plan: specify why a test is TODO.
- TODO
:todo, :todo(1)
Sometimes, it just doesn't fit anywhere else, so in that case it is just TODO.
So, vanilla.
- TODO bug
:todo<bug>, todo => 'bug', :todo('bug')
This is either an outright bug in Pugs, or it is a feature which /should/ be implemented, but for some reason (usually some kind of seemingly unrelated bug) it is not. This is a good "layer" for edge cases tests and other such ugly corners.
So this is a test which should be working, but isn't. Still, the definition for this one is a little hazy in my mind.
This one is a little hazy for me too. Obviously if it is a bug, then we label it as such (most of these are in t/pugsbugs), but sometimes there are edge cases which just have not been fleshed out enough, and this is where this attr comes in.
And to be honest, I think it is okay that it is a little hazy, as the reason for the failure may not always be immediately obvious to the test developer, but a lambda-folk later on may know exactly what the issue is. So in a way this category almost implies a certain amount of ambiguity (especially when found outside t/pugsbugs).
- TODO feature
:todo<feature>, todo => 'feature', :todo('feature')
These are tests written for perl6 features which are not yet implemented in Pugs. Objects is the perfect example of this.
This is what TODO is all about: this is going to be implemented, but isn't yet, so I'm marking it TODO until it is implemented.
So maybe the 'bug' one means: this was implemented, but now seems to be unimplemented (because internals changed somehow, etc.).
Yes, internals change so often lately (with Autrijus's speed-ups and such) and they are likely to change more as the OO model gets put in too.
- TODO parsefail
:todo<parsefail>, todo => 'parsefail', :todo('parsefail')
These are tests which fail to parse correctly. These are usually tested
with eval_ok() or eval_is().
So this is like 'feature', except that the parser does not understand the syntax yet. Maybe it could be called 'syntax', which might make more sense.
Sometimes it is a 'feature', other times it is just some odd combination/ordering that the parser cannot comprehend. These types of TODOs tend to not stay around for very long either.
- TODO hardfail
:todo<hardfail>, todo => 'hardfail', :todo('hardfail')
These are tests which parse okay, but die with an uncatchable error. I will be experimenting with using the "is lazy" trait to catch and not-evaluate these (and therefore not kill pugs).
This one is hazy in my mind, too. I'm guessing it means the syntax
parses, but is still not understood, so the interpretter dies when
trying to execute. It's the difference between understanding (the
syntax) and comprehending (the meaning). English doesn't really have
good words to describe it. Dutch uses 'verstaan' (for words/syntax) and
'begrijpen' (for comprehension/meaning).
This is one Autrijus suggested. It is basically this:
Pugs can parse the code. Pugs can execute the code. An error is thrown which Pugs cannot catch.
There is still a difference between Pugs-catchable errors and non-Pugs-catchable errors. As I said too, I am going to experiment with the "is lazy" trait and this attr to see if we cannot avoid Pugs dying.
My goal was to support these different kinds of TODOs with Test.pm, and
also make them visible through Test::Harness (and eventually in our
smoke test). In exploring Test::Harness I noticed that TODO tests can
also have a "reason", just like SKIP tests do (although Test::Harness
seems to ignore them) and I have exploited this feature to accomplish
my goal.
Very nice.
So currently (in the latest revision), Test.pm will support the following code:
ok(bad_function(), "... testing bad_function", :todo<bug>);
this will produce the following TAP output:
not ok 1 - ... testing bad_function # TODO bug # Failed (TODO bug) test (-e line 3, column 1-46) # Got: undef
I have also altered our local copy of Test::Harness (in inc/) so that if you were to run this with 'prove' or 'make test' you would see something like this:
t/my_test_file....ok 1/5 TODO bug test 2/5 TODO tests 2/5 TODO feature tests All tests successful.
This allows a simple top-level overview of not only the number of TODO tests, but of each specific TODO test "layer". Ideally we can support this level of TODO granularity in the test smoke interface as well.
That looks really useful. We might just be able to leave the TODO tests
as TODO, once we have all the reasons in place.
Well once they succeed, they will show up as unexpected successes.
Now all that is left is for people to start changing all the :todo tests to :todo<some_attribute>.
Can you put the definitions of each attribute in one of the README files?
I will put these common ones in there yes. However, the nice aspect of using the TODO "reason" attribute is that I am not limiting us to a strict sub-set of TODO attributes. So if you so desire, you can make them up on your own (or write them in Dutch :)).
As always comments, question and suggestions are always welcome.
And gladly given. Good work, Stevan.
Danke :)
And thanks for the comments as well.
Stevan
