>>>>> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL PROTECTED]> 
>>>>> said:

 >> Asking the wrong question. None of our testsuites is there to protect
 >> against spoof or attacks. That's simply not the goal. Same thing for
 >> 00-signature.t

  > We would seem to be agreeing.  If the goal of the test suite is not to 
  > against spoofing, and if it doesn't accomplish that anyway, why put a
  > signature check in there?

Of course we are agreeing 99%. But I'm citing the Michael Schwern
saying that is dearer to me than the above paragraph: tests are there
to find bugs.

 >> Yupp. And testing the signature in a test is better than not
 >> testing it because a bug in a signature or in crypto software is
 >> as alarming as a bug in perl or a module.

  > I believe this to be outside the scope of a given module's tests.  It's not
  > the responsibility of every CPAN module to make sure that your crypto 
  > is working.  Or perl.  Or the C compiler.  Or make.  That's the job of the
  > toolchain modules which more directly use them (CPAN, Module::Signature,
  > MakeMaker, Module::Build, etc...). [1]

You're barking up the wrong tree. Stick to your older wisdom that bugs
are there to find tests, err, forgive me, tests are there to find
bugs, and when a test found a bug it is a hero forever.

  > At some point you have to trust that the tools work, you can't test the 
  > universe.  You simply don't have the time.

I agree totally. But you also do not have the wisdom that you can
predict which tests will find bugs in which software. Your test found
a bug in Module::Signature? Great, so it was a good test.

Of course you should not annoy people with a test once the bug is
known and reported. Make a new release, skip the test with the skip
statement 'Module::Signature 123.45 known bug see RT #54321' and wait
for the next time the test finds a bug once Module::Signature gets

Besides that I agree with the rest of your musings. Great writup. Some
guidelines about cost/benefit relations are worthwhile but one should
take care not to make them too narrow. The last thing we want to have
is a new kind of CPAN police dictating people to remove tests.

  > [...] But if the test failed because it's a bad test,

Clearly a strawman's argument. It's impossible to contradict you on
this. Thou shalt not write bad tests. Period.

  > Let's look at the example of Test::More.  The last release has 120 passes 
  > just 4 failures.
  > http://cpantesters.perl.org/show/Test-Simple.html#Test-Simple-0.74

  > What are those four failures?  Three are due to a threading bug in certain
  > vendor patched versions of perl, one is due to the broken signature test.

  > Look at the previous gamma release, 0.72.  256 passes, 9 failures.
  > 5 due to the threading bug, 4 from the signature test.

  > 0.71:  73 passes, 2 failures.  1 signature, 1 threads

  > 0.70:  221 passes, 12 failures.  3 signature, 9 threads

  > And so on.  That's nine months with nothing but false negatives.

The bug lies in the time span of nine months. Bad tests are bugs,
insidious bugs and need to be fixed and shall not be kept for 9

  > The
  > signature test is not actually indicating a failure in Test::More, so it's 
  > no benefit to me or the users, and the bug has already been reported to
  > Module::Signature.

See above. Once the bug is reported there is no justification to keep
the test around. In this case I prefer a skip over a removal because
the test apparently once was useful.

  > The threading test is indicating a perl bug that's very difficult to detect
  > [2], only seems to exist in vendor patched perls, I can't do anything about
  > and is unlikely to effect anyone since there's so few threads users.  It's
  > already been reported to the various vendors but it'll clear up as soon as
  > they stop mixing bleadperl patches into 5.8.

  > In short, I'm paying for somebody else's known bugs.  I get nothing.
  > Test::More gets nothing.  The tools get nothing.  Cost with no benefit.  So
  > why am I incurring these costs?  Maybe the individual users find out their
  > tools are broken, but it's not my job to tell them that.

During smoking CPAN I often find bugs in one module revealed by a test
in another one. Only because David Golden tests so hard his tests were
well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask
if it is his job or not. Just a few RT headlines of the past year:
Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication.
DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09
broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem.
Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE.
DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke
Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor
breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI.
File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02
breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks
SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's
NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks
Verilog::Language. And so on.

  > I've kept in the threading test because the perl bug it's tickling
  > does have a direct effect on Test::More, and it could indicate
  > future threading issues, but lacking any way to resolve it I'm
  > tempted to pull it.

  > The signature test, otoh, does not indicate anything that effects
  > Test::More.

Sure it does. There is a symbiotic relation between modules on CPAN.
Weigh that in and we're fine.

  >  The ability or inability to check the signature has nothing to do
  > with the operation of Test::More. So why am I checking it? The
  > Test::More test suite isn't a full service gas station. It's not
  > going to wash the windows and check the oil and give you
  > directions. It makes sure Test::More works and that's that.

  > As you can see, this is a considered analysis. In general,
  > redundant tests are ok. They're often not truly redundant but just
  > have a large overlap. And this all assumes the tests have some
  > benefit.

  > And often it's more trouble than it's worth to ferret out test
  > redundancies. Worse yet is the creeping mental attitude of "should
  > I write this test? Can I justify the cost?" This is like the
  > related attitude of "can I justify cleaning up this code?"
  > Unchecked, both lead to paralysis.

  > But in this case it's a test which has no benefit to the module,
  > [for the signature test] indicates no failure of module
  > functionality, is reporting on known bugs in other tools and is a
  > test that should be and is done by other modules more directly.
  > Cost and redundancy with no benefit.

Maybe you want to borrow a bit from my Module::Signature test in
CPAN.pm. It didn't find a bug in Module::Signature but it is well
suited to find other bugs that would be really relevant for me and my
users. I'll keep mine in and I hope others do likewise. And I hope
nobody tells me to remove working tests.


Reply via email to