Andreas J. Koenig wrote: >>>>>> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL >>>>>> PROTECTED]> said: > > We would seem to be agreeing. If the goal of the test suite is not to > protect > > against spoofing, and if it doesn't accomplish that anyway, why put a > > signature check in there? > > Of course we are agreeing 99%. But I'm citing the Michael Schwern > saying that is dearer to me than the above paragraph: tests are there > to find bugs.
I say lots of apparently contradictory things. The trick is knowing when one rule wins out over the other. Something to keep in mind is that I'm talking about one very specific test. Don't let this discussion get tangled up in the "author tests" brouhaha that often brews up around here. > > [...] But if the test failed because it's a bad test, > > Clearly a strawman's argument. It's impossible to contradict you on > this. Thou shalt not write bad tests. Period. That was supposed to come out more like "if the test failed because of a mistake in the test suite". You know the sort of thing. Like when you write: like $error, qr/your shit is broke at $0 line \d+\.\n/; and it blows up on Windows because you forgot about the backslashes in Windows path names. The test failure indicates a bug in the test, not the code. Thus, the failure has a cost and no benefit. > > The > > signature test is not actually indicating a failure in Test::More, so > it's of > > no benefit to me or the users, and the bug has already been reported to > > Module::Signature. > > See above. Once the bug is reported there is no justification to keep > the test around. In this case I prefer a skip over a removal because > the test apparently once was useful. Buuuut skipped tests don't get run so it's effectively deleted, except a permanently skipped test sits around cluttering things up. Smells like commenting out code that maybe someday you might want to use again in the future. Just adds clutter. If I want to bring a test (or code) back from the dead that's what version control is for. > > The threading test is indicating a perl bug that's very difficult to > detect > > [2], only seems to exist in vendor patched perls, I can't do anything > about > > and is unlikely to effect anyone since there's so few threads users. It's > > already been reported to the various vendors but it'll clear up as soon as > > they stop mixing bleadperl patches into 5.8. > > > In short, I'm paying for somebody else's known bugs. I get nothing. > > Test::More gets nothing. The tools get nothing. Cost with no benefit. > So > > why am I incurring these costs? Maybe the individual users find out their > > tools are broken, but it's not my job to tell them that. > > During smoking CPAN I often find bugs in one module revealed by a test > in another one... Only because David Golden tests so hard his tests were > well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask > if it is his job or not. Just a few RT headlines of the past year:... > Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication. > DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09 > broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem. > Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE. > DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke > Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor > breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI. > File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02 > breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks > SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's > NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks > Verilog::Language. And so on. I agree that's all very useful. Interlocking dependent test suites ferret out bugs the original authors wouldn't find. However, there is a very important difference between the list above and Test::More's signature test. On a quick scan, all of those modules have direct dependencies. DBD::Sybase uses DBI, Lingua::Stem uses Test::Distribution, etc... so it's natural that their tests would test their dependencies. If a dependency breaks, they break. I'm sure most of the authors above did not set out with the intention to test their dependencies, it's all inherent in the testing of their own code. Test::More doesn't actually use Module::Signature, so why is it testing it? It would be like if DBI decided to add a test to make sure MakeMaker can read the MANIFEST. Sure it's useful to know that part of the toolchain works and that the MANIFEST can be read, but why is that in DBI? One can argue that DBI depends on the good functioning of ExtUtils::Manifest to install, so it should test it. Ok, then what about all the other things DBI depends on to install? Should it test that MakeMaker can make a valid Makefile? Should it test that tar and gzip work? Should I check that CPAN.pm can properly download and install modules? Should it check that Test::Harness can parse TAP? That Test::More is outputting correct TAP? At some point you have to draw the line in how much of the universe you're going to be responsible for. I like to put that border at the edge of the module's functionality, it has a nice, sharp edge and it lets a module author do their thing without going insane. The signature test goes past that line so I'm going to clip it off. > Maybe you want to borrow a bit from my Module::Signature test in > CPAN.pm. It didn't find a bug in Module::Signature but it is well > suited to find other bugs that would be really relevant for me and my > users. I'll keep mine in and I hope others do likewise. And I hope > nobody tells me to remove working tests. CPAN.pm is different. It relies on Module::Signature for it's features. So it makes perfect sense for CPAN.pm to test Module::Signature. PS Irony of ironies, just after I wrote my previous reply I got a bug report about that fucking thread test again. -- Hating the web since 1994.