On 17/09/06, chromatic <[EMAIL PROTECTED]> wrote:
On Saturday 16 September 2006 14:42, Fergal Daly wrote:
> So, a passing TODO tells me that either the author
> - writes bad tests
> - means something else entirely by TODO
> - didn't run the test suite before this release
I can buy these, to some degree.
> or
> - my environment is different to the authors in a way that has an
> impact on the workings of the module
This is the one that bothers me. Ignoring the issue that this is Really Not a
Job for the Parser, I'm uncomfortable effectively recommending that people
*not* use a module until someone deliberately reports "Hey, it works better
than you think on this platform!" to the author and the author releases a
completely new version to un-TODO those tests.
I agree that it's not the paser's job to interpret the results, that's
for the harness.
As for this case, having thought about it a bit more, I don't even
accept it as a legitimate use of TODO. If you don't support a platform
(or you don't support platforms that do X) then you test for X and
then SKIP.
So while I don't think I'd vote for stopping the install because of an
expected pass, I don't think it's ever a good sign. The idea of
something working "better" than than the author expected is a bit
dubious.
TODO: {
local $TODO = "archiving not implemented yet";
$archive = $mailbox->archive;
ok(-z $mailbox->filename, "truncated mailbox");
ok(-e $archive, "archive file has been created");
}
If I get
OK 1
NOT OK 2
is that working better than expected?
F
I don't want to get in the habit of telling installers to ignore test
failures, because that is Just Bad.
I don't want to get in the habit of making busywork for authors, because I
don't want to do it myself, I don't want them to skip adding legitimate and
useful tests, and I don't want them to skimp on tests where a TODO test is
appropriate.
I don't want someone to suggest maintaining external test metadata about
expected platforms and expected skips and expected TODOs because hey, look
over there, that's overengineering something that could be simple.
If and only if bonus tests actually *harm* the use of the module should it be
a possibility that they cause an error condition. I've seen a lot of
hypothetical situations, but how about real examples of TODO tests that
actually exist? I know how Schwern and Ian and I have documented them in
various forms; how do people actually use them?
(Also, again, Not The Parser's Job.)
-- c
