On Sun, Jul 29, 2001 at 08:09:13AM -0400, Barrie Slaymaker wrote:
> The "shouldbe" tests would be like "todo" tests in that their ok/not ok
> status doesn't affect the "All tests successful" result, but the
> expectation is that they should pass; the opposite of the todo test.
> The idea being that if I want a quick read from CPAN testers or the
> general public on whether (or how widely ) a construct I want to depend
> on is broken on different platforms, I could use a "shouldbe" test that
> would be quiet on success and harmlessly noisy on failure (again, the
> opposite of todo()).
Oh, this came up at the Testing BoF at TPC. The context was
LWP::RobotUA, whose tests never seem to pass. They were thinking of
allowing a set of tests which might fail but would be ignored.
I don't like that idea. At least, the idea of making Test::Harness
officially recognize it.
Currently, tests are binary. Either it passes or it does not. This
makes it very simple to know if things are or are not working. Adding
in fuzzy tests that may or may not work moves us away from that.
More importantly, when a test says its ok, its OK. Everything has
passed, and there are no fuzzy failures. This builds confidence in
the testing suite. I spoke of the psychological phenomenon of "one
broken window". You abandon a car on the side of the road, and it
will lie untouched for days. You smash one window and it will be
stripped in hours. A single unrepaired failure breeds a sense of
abandonment and rapidly leads to further failures and rapidly the
whole system breaks down. I've seen this in action.
Having a test who's outcome is not important to the success or failure
of the test is that one broken window. Since it does not have to be
fixed, it will not be fixed. Worse, rather than repairing legitimate
failures, people may be inclined to turn them into "shouldbe" tests.
This doesn't mean you couldn't simply print a message to STDERR.
Test::Harness doesn't catch STDERR and it will be seen by CPAN testers
who are, in general, a paranoid bunch.
warn "# $foo should be $bar! Please report this to the author."
unless $foo eq $bar;
--
Michael G. Schwern <[EMAIL PROTECTED]> http://www.pobox.com/~schwern/
Perl6 Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One
Teleportation must be invented. If we don't invent teleportation, China will
throw nuclear bomb everywhere. Especially now everyone can live forever.
--Alex Chiu, Immortality Guy
