Re: passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))
On Fri, Sep 05, 2008 at 04:07:43PM +0100, Nicholas Clark wrote: On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote: Careful attention and responsiveness to CPAN testers and to rt.cpan.org is the best cure for this. Alternatively, I could just not upload code to CPAN, and not have this problem. You're right that it's a problem. I'm not convinced that your cure will work with all real world volunteers. The best cure is that if someone wishes to use a piece of software and finds that it has gone rotten, he can take over maintenance of it. That's how I ended up with Data::Compare. -- David Cantrell | Godless Liberal Elitist Aluminum makes a nice hat. All paranoids will tell you that. But what most do not know Is reflections will show On the CIA's evil landsat.
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Fri, Sep 05, 2008 at 11:36:10AM -0700, chromatic wrote: It's a little bit like trying to have a discussion with someone who's upset but won't tell you why, and you have to guess and hope you don't make things worse before you get a useful answer. Thankfully, unlike in personal relationships, politely asking a CPAN-tester WTF? won't get you slapped. -- David Cantrell | London Perl Mongers Deputy Chief Heretic In this episode, R2 and Luke weld the doors shut on their X-Wing, and Chewbacca discovers that his Ewok girlfriend is really just a Womble with its nose chopped off.
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
In article [EMAIL PROTECTED], Dave Rolsky [EMAIL PROTECTED] wrote: ThatÄôs CPANTS, the useful part of it anywayĶ except that the feedback cycle is even *much* slower there than with the Testers. Is not! Check out Test::Kwalitee on cpan. I've taken to adding that to my maintainer-only tests for my distros. And you can run cpants_lint.pl yourself too. Module::Release now has that built in as well.
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Fri, 5 Sep 2008, Aristotle Pagaltzis wrote: * chromatic [EMAIL PROTECTED] [2008-09-05 19:50]: If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. That’s CPANTS, the useful part of it anyway… except that the feedback cycle is even *much* slower there than with the Testers. Is not! Check out Test::Kwalitee on cpan. I've taken to adding that to my maintainer-only tests for my distros. -dave /*== VegGuide.Org Your guide to all that's veg ==*/
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thursday 04 September 2008 15:21:19 David Cantrell wrote: What I'm not willing to do, however, is to manually check every report and ensure perfection that way. Why? Because it takes too long, and I have a job and a life. And anyway, I'd still make mistakes - and even if I don't make mistakes, people will still think I have. Everyone makes mistakes when they're doing a boring job, doubly so without the prospect of reward. So anyone who insists that I read every report I send to them will just get no reports from me. With respect, the people receiving and manually checking those unsolicited, automated, opt-out bulk reports also have jobs and lives. -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 04, 2008 at 09:50:01AM -0700, chromatic wrote: On Thursday 04 September 2008 01:19:44 Eric Wilhelm wrote: Let's pretend that I'm a real jerk of an author and I only care about whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky vendor patches) with a fully updated and properly configured toolchain and the clock set to within 0.65s of NTP time. Oh great, yet another check I have to add to my Build.PL. What's the magic cantrip for THAT one? (Why yes, I *have* seen bugs related to time skew on network-mounted paths), So have I. I think that there's at least one stat test in the core that will fail if you're testing on an NFS mount from a machine where the clock differs. IIRC it's because file creation time is stamped by the server, and file modification time is stamped with a time from the client, and if they're out the impossible can happen. Which when you're sanity testing the ops that read these sorts of things, you're looking out for. Although I suspect that there can be more general problems with make getting (correctly) confused by timestamps on files it touched. Nicholas Clark
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
* Andy Lester [EMAIL PROTECTED] [2008-09-04 17:45]: Who's to say what my job as an author is? No one, but at the same time, you as an author of libre software have no moral right to dictate what your users want from your code, and if your job according to your understanding does not extend to satisfying the users’ expectations according to *their* understanding, then the users have a legitimate case for knowing that your distributions are made of FAIL as far as they are concerned. Whether you choose to care is then your prerogative, obviously. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))
On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote: I recognize that CPAN is a volunteer effort, but it does seem to me there is a implicit responsibility on the part of the author to maintain the module going forward, or to pass the baton to someone else. Call it a Best Is there an easy central visible way to flag up a module as up for adoption? What should have been the right list to ask that question on? Practice, if you will. The end-user simply wants the module to work. Maintainers not paying attention, and the subsequent bitrot that is appearing on CPAN, is one of Perl's biggest problems at the moment. Careful attention and responsiveness to CPAN testers and to rt.cpan.org is the best cure for this. Alternatively, I could just not upload code to CPAN, and not have this problem. You're right that it's a problem. I'm not convinced that your cure will work with all real world volunteers. Nicholas Clark
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 01:19, Eric Wilhelm wrote: But with the per-tester direct mail, the recipient is powerless to stop it, and feeling powerless tends to make people angry. This, to me, demonstrates better than most points how CPAN Testers is being crushed by its own success. A few years ago, I got very few reports, so it was no big deal. But nowadays, when I release a module, I can expect it to be tested a *lot* in the 24-36 hours after release. If I fucked something up, that can be a lot of FAIL mails appearing in my inbox. Inspiration: So for CPAN testers site 404s, we should display a FAIL MAIL. Kinda like the fail whale and the fail snail. :-P Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 10:09, chromatic wrote: My job is editor, not programmer. Also novelist -- but again, not programmer. Certainly not CPAN programmer. What's your novel? Can I read it? Paying attention is not my job. Releasing software I've written under a free and open license does not instill in me the obligation to jump whenever a user snaps his or her fingers. Well, you can ignore the FAILs. Or you can evaluate each one to determine if you could change something your code to make it easier for your users. No one compels you to do anything. I may do so because I take the quality and utility of my software seriously, but do not mistake that for anything which may instill in you any sort of entitlement. That is an excellent way not to get what you want from me. I don't think anyone would argue that. Straw man, dude. I don't like this: failure by any other name would smell just as bad. In other words, if an end user is not going to have a happy, functional module after typing install Foo::Bar at the CPAN prompt, this is a failure that should be noted as such and fixed by the author. Then CPAN Testers reports should come with login instructions so that I can resurrect decade-old skills and perform system administration to fix broken installations and configurations -- oh, and as you say, a truly *modern* reporting system should publish these logins and passwords publicly in syndication feeds. Huh? I don't think he's referring to configuration issues on the tester's box. Clearly that's not the author's responsibility. It's the job of CPAN Testers to try to minimize the FAIL reports for such a situation, but not your job to change anything when the occasional invalid FAIL gets through. That's not to say that CPAN Testers couldn't suggest a way for you as an author to cut down on those FAILs, but you're not compelled to do anything. However, by what possible logic can you conclude that the appropriate way to get that bug fixed is to report it to people who, given all of the information detected automatically, *do not* maintain CPAN.pm? Obviously it's not always easy to identify the source of the bug. It is the responsibility of CPAN Testers to run the most recent module in order to minimize such circumstances. Oh, perhaps you think, it's easy for them to read the reports and diagnose the problem remotely on machines they have never seen before, did not configure, and cannot probe -- and it's so easy for them to file a bug in the right place! If you don't think that, precisely what *do* you think to produce such a bold assertion that it is Shlomi's job to install and reconfigure a new version of CPAN.pm for a CPAN Tester -- or for that matter, everyone with a misconfigured version of CPAN.pm which contains this bug? Straw man again. When I get such a report, I email the CPAN Tester and say, WTF? He usually gets back to me with, My bad, sorry, won't happen again. It's not my job to fix bugs in *my own* distributions. I do it because I care about quality and, contrary to what appears to be near- universal belief around here, I care that people can use my code. Straw man again. Do you really believe anyone is actually saying that? Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 10:50, David Cantrell wrote: Change the record, please. This one's getting boring. Maybe I should start being equally loud and obnoxious about obviously stupid and broken things like the existence of UNIVERSAL-isa. It might give you some appreciation for how you're coming across here. Let's keep it civil, shall we? David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 15:21, David Cantrell wrote: What I'm not willing to do, however, is to manually check every report and ensure perfection that way. Why? Because it takes too long, and I have a job and a life. How about checking a random sample of them, just as a sanity check for your toolchain? Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 11:53, Andy Lester wrote: Maybe what's so frustrating to me, and perhaps to chromatic, and whoever else ignores CPAN Testers but doesn't discuss it, is that we're being fed things that we should be thankful for and goddammit why aren't we appreciative??!? Here are the things that we have determined are quality. Here are test reports reporting on failures for these things that we care about you caring about. Again, this is CPANTS, not CPAN Testers. Maybe you should add a co-maintainer. Your responsibility as an author is to... These are not in test reports. They were in this thread. And they're suggestions to you. Do what you want with them. There is far too much bile floating in this thread considering that I believe we all have a shared interest in the quality of our code. Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Here are test reports reporting on failures for these things that we care about you caring about. Again, this is CPANTS, not CPAN Testers. Getting failure reports for a module not running on Perl 5.005 is a test about something I don't care about. I don't give Shit One if my code runs on 5.005, and yet, I've had failures for them. Maybe you should add a co-maintainer. Your responsibility as an author is to... These are not in test reports. They were in this thread. And they're suggestions to you. Do what you want with them. And yet they're encapsulations of this entire problem. It's unsolicited advice. You should make your code handle 5.005. You should have checks for such-and-such a platform. Who is anyone to say? There is far too much bile floating in this thread considering that I believe we all have a shared interest in the quality of our code. That quality slider is long and multidirectional. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 09:13, Andy Lester wrote: Here are test reports reporting on failures for these things that we care about you caring about. Again, this is CPANTS, not CPAN Testers. Getting failure reports for a module not running on Perl 5.005 is a test about something I don't care about. I don't give Shit One if my code runs on 5.005, and yet, I've had failures for them. Well, yeah, I have too. And sometimes I make a tweak to get things working on 5.005, and other times I tell my users that it runs 5.006 or later by saying so in Build.PL. Seems reasonable to me to specify such dependencies. These are not in test reports. They were in this thread. And they're suggestions to you. Do what you want with them. And yet they're encapsulations of this entire problem. It's unsolicited advice. You should make your code handle 5.005. You should have checks for such-and-such a platform. Who is anyone to say? No one says that. They say, Hey, this fails on 5.05. No one is suggesting what you should do about it. That's your choice. Of course, those of us interested in quality hope you'll tell your users about a Perl version dependency, but no one is demanding anything. There is far too much bile floating in this thread considering that I believe we all have a shared interest in the quality of our code. That quality slider is long and multidirectional. That doesn't make it any less real. Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Well, yeah, I have too. And sometimes I make a tweak to get things working on 5.005, and other times I tell my users that it runs 5.006 or later by saying so in Build.PL. Seems reasonable to me to specify such dependencies. Seems reasonable to me is EXACTLY my frustration. That is YOUR standard that YOU are specifying. How about if I start sending email to everyone, whether they want it or not, if their code doesn't run under -T checking? It seems reasonable to me that it should. those of us interested in quality hope you'll tell your users about a Perl version dependency, but no one is demanding anything. And you will spam me if I don't provide that dependency. This is beyond me and my frustration about getting worthless emails. It is about the presumption of telling CPAN authors what they can upload. No, we're not preventing people from uploading anything, but we're punishing them for not bending to the whims of the CPAN Testers ideals. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 09:34, Andy Lester wrote: Well, yeah, I have too. And sometimes I make a tweak to get things working on 5.005, and other times I tell my users that it runs 5.006 or later by saying so in Build.PL. Seems reasonable to me to specify such dependencies. Seems reasonable to me is EXACTLY my frustration. That is YOUR standard that YOU are specifying. Well, how else do your users know what versions of Perl you support? How about if I start sending email to everyone, whether they want it or not, if their code doesn't run under -T checking? It seems reasonable to me that it should. That's different, isn't it? Your code makes no claims about external dependencies. You can control that. Making code run under -T might be a decent Kwalitee metric, but it has nothing to do with whether or not your tests pass. You can control that in your tests by putting -T on the shebang line of individual test scripts. those of us interested in quality hope you'll tell your users about a Perl version dependency, but no one is demanding anything. And you will spam me if I don't provide that dependency. Spam is in the eye of the beholder, I guess. David Golden won't send you any reports anymore, and I completely agree that email reports should be opt-in. This is beyond me and my frustration about getting worthless emails. It is about the presumption of telling CPAN authors what they can upload. No one is telling CPAN authors what they can upload. No, we're not preventing people from uploading anything, but we're punishing them for not bending to the whims of the CPAN Testers ideals. Punishing? Punishing would be removing a module from CPAN. Getting fail report emails is annoying and should be changed to be opt-in. Would that solve your problem? Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
* Andy Lester [EMAIL PROTECTED] [2008-09-05 18:15]: Here are test reports reporting on failures for these things that we care about you caring about. Again, this is CPANTS, not CPAN Testers. Getting failure reports for a module not running on Perl 5.005 is a test about something I don't care about. I don't give Shit One if my code runs on 5.005, and yet, I've had failures for them. Sure. You have had failures because your code doesn’t run on 5.5 means it doesn’t run on 5.5 means it doesn’t run on 5.5. If I am using 5.5 for whatever reason and am considering trying your code for whatever reason [chromatic: let’s not get into that right here], then to me it is interesting to know that your code is going to fail. I don’t give Shit One about how you feel about me running 5.5, I just want to know if the code works or not. I would be interested to know that you don’t care about supporting my configuration, but as you don’t even care enough to declare your non-support explicitly, I have to find out otherwise. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Friday 05 September 2008 08:48:36 David E. Wheeler wrote: On Sep 4, 2008, at 10:09, chromatic wrote: Well, you can ignore the FAILs. Or you can evaluate each one to determine if you could change something your code to make it easier for your users. No one compels you to do anything. You're right, there's no compel. If reports don't come by email to people who haven't asked for them, then they'll only get reported via an RSS feed I can choose to read or not, and on the search.cpan.org pages of my distributions, which I don't visit. That's not compulsion, that's just offering public data points in yet another location I can choose to visit or disregard. Thus now I can get bug reports via: * personal mail * RT web/RSS * CPAN Forum * CPAN Testers (email/RSS) * CPAN Annotations * CPANTS * Usenet * PerlMonks * countless mailing lists I may have missed a few. In my ideal world, CPAN Testers would be a series of platform combinations where I could submit a development version of a distribution and get test results with the latest version of dependent modules and toolchain modules *before* uploading to the CPAN. * It's opt-in * It answers my most important question when I want it answered * It can provide a reference configuration for CPAN installers * It gives me something I don't already have * It focuses tester resources where (I believe) they're most appropriate I may do so because I take the quality and utility of my software seriously,but do not mistake that for anything which may instill in you any sort of entitlement. That is an excellent way not to get what you want from me. I don't think anyone would argue that. Straw man, dude. I refer you to Greg Sabino Mullane's posts, in which the gentlest expectation is: I recognize that CPAN is a volunteer effort, but it does seem to me there is a implicit responsibility on the part of the author to maintain the module going forward, or to pass the baton to someone else. Call it a Best Practice, if you will. The end-user simply wants the module to work. Maintainers not paying attention, and the subsequent bitrot that is appearing on CPAN, is one of Perl's biggest problems at the moment. Careful attention and responsiveness to CPAN testers and to rt.cpan.org is the best cure for this. However, by what possible logic can you conclude that the appropriate way to get that bug fixed is to report it to people who, given all of the information detected automatically, *do not* maintain CPAN.pm? Obviously it's not always easy to identify the source of the bug. It is the responsibility of CPAN Testers to run the most recent module in order to minimize such circumstances. That would increase the utility of CPAN Testers to me if it were true (at least, if by most recent module you mean toolchain modules). It's not my job to fix bugs in *my own* distributions. I do it because I care about quality and, contrary to what appears to be near- universal belief around here, I care that people can use my code. Straw man again. Do you really believe anyone is actually saying that? I can't find the link, but someone here on Wednesday said You don't care if people actually use your code, and it's not the first time I've heard it. -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Friday 05 September 2008 10:31:29 Aristotle Pagaltzis wrote: I would be interested to know that you don’t care about supporting my configuration, but as you don’t even care enough to declare your non-support explicitly, I have to find out otherwise. I don't like the check testers/grumble/upload new distribution with no functional changes just niggly little packaging bits you hope will opt out of testers tests you don't care about/sleep/repeat cycle. It's a slow, clunky black box game where the rules aren't always clear and you have to upload new versions of your distributions that don't do much for most of your users. I would find some sort of *optional* distribution packaging best practices scanner more useful here than getting CPAN Testers reports (by whatever mechanism). If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. -- c
Re: passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))
In article [EMAIL PROTECTED], Nicholas Clark [EMAIL PROTECTED] wrote: On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote: I recognize that CPAN is a volunteer effort, but it does seem to me there is a implicit responsibility on the part of the author to maintain the module going forward, or to pass the baton to someone else. Call it a Best Is there an easy central visible way to flag up a module as up for adoption? What should have been the right list to ask that question on? A couple of the PAUSE admins have been talking about that, but we haven't really decided anything about how it should happen. There would probably be some virtual PAUSE ID that people could pass primary maintainership too and once those modules are there, someone could request maintainership of them without a waiting period. That's the way that would work with what is already in place, although someone has to upload a new dist for it to show up in the new account. I was thinking we'd want to do that anyway to at least modify the docs to note its status. Or, Andreas could change PAUSE, which is a bit more involved :)
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 10:32, chromatic wrote: You're right, there's no compel. If reports don't come by email to people who haven't asked for them, then they'll only get reported via an RSS feed I can choose to read or not, and on the search.cpan.org pages of my distributions, which I don't visit. That's not compulsion, that's just offering public data points in yet another location I can choose to visit or disregard. Thus now I can get bug reports via: * personal mail * RT web/RSS * CPAN Forum * CPAN Testers (email/RSS) * CPAN Annotations * CPANTS * Usenet * PerlMonks * countless mailing lists And if you have to opt-in, I imagine that would solve the biggest complaint, yes? It's the unsolicited email reports that are annoying, right? In my ideal world, CPAN Testers would be a series of platform combinations where I could submit a development version of a distribution and get test results with the latest version of dependent modules and toolchain modules *before* uploading to the CPAN. Well, you can upload a dev version to CPAN and the testing bots will test it, I believe. It'd be nice if there was a separate place to upload code to be tested before you actually released it. That'd be very handy indeed. * It's opt-in * It answers my most important question when I want it answered * It can provide a reference configuration for CPAN installers * It gives me something I don't already have * It focuses tester resources where (I believe) they're most appropriate Yeah, it's a reasonable feature request. Will probably have to wait for the Web service, though. I may do so because I take the quality and utility of my software seriously,but do not mistake that for anything which may instill in you any sort of entitlement. That is an excellent way not to get what you want from me. I don't think anyone would argue that. Straw man, dude. I refer you to Greg Sabino Mullane's posts, in which the gentlest expectation is: I recognize that CPAN is a volunteer effort, but it does seem to me there is a implicit responsibility on the part of the author to maintain the module going forward, or to pass the baton to someone else. Call it a Best Practice, if you will. The end-user simply wants the module to work. Maintainers not paying attention, and the subsequent bitrot that is appearing on CPAN, is one of Perl's biggest problems at the moment. Careful attention and responsiveness to CPAN testers and to rt.cpan.org is the best cure for this. Nothing to do with entitlement. I have hopes for the quality of code, and hope that you'll maintain it, but I don't feel entitled to it. Obviously it's not always easy to identify the source of the bug. It is the responsibility of CPAN Testers to run the most recent module in order to minimize such circumstances. That would increase the utility of CPAN Testers to me if it were true (at least, if by most recent module you mean toolchain modules). Yes, I do. And the main CPAN testers do try to stay up-to-date. But getting them all to do so is of course an uphill battle. But certainly those who use bots tend to stay on top of things, IME. It's not my job to fix bugs in *my own* distributions. I do it because I care about quality and, contrary to what appears to be near- universal belief around here, I care that people can use my code. Straw man again. Do you really believe anyone is actually saying that? I can't find the link, but someone here on Wednesday said You don't care if people actually use your code, and it's not the first time I've heard it. Yeah, people can be dicks on mail lists (/me raises his hand), but you don't get that in CPAN Testers reports, do you? You can ignore the assholes. I (usually) do (and FWIW, I didn't notice anything like that in this thread, and David Golden apologized for his out-of-line comment). Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 10:46, chromatic wrote: I don't like the check testers/grumble/upload new distribution with no functional changes just niggly little packaging bits you hope will opt out of testers tests you don't care about/sleep/repeat cycle. It's a slow, clunky black box game where the rules aren't always clear and you have to upload new versions of your distributions that don't do much for most of your users. Yeah, but that's a symptom of the voluneerism of the organization. We don't have a dedicated serve farm you can upload a potential distribution to and get results in an hour. It'd be nice to have, but uploading development releases to CPAN and waiting 24-48 hours is as close as it gets right now. I would find some sort of *optional* distribution packaging best practices scanner more useful here than getting CPAN Testers reports (by whatever mechanism). If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. Not a bad idea. Surely someone could write a test (or Perl Critic plugin) for that, yes? Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 12:24 PM, David E. Wheeler wrote: Punishing? Punishing would be removing a module from CPAN. Getting fail report emails is annoying and should be changed to be opt-in. Would that solve your problem? One person's annoying is another person's punishment. The key here is that my reward for uploading to CPAN is to get mass unsolicited email, unless my upload conforms to some arbitrary standard that I was not informed of in advance. Where's the warning on the front page of PAUSE saying Your upload had better match certain criteria or else we will send you mass email about it? Am I the only one looking at this from the point of view of others? xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 12:24 PM, David E. Wheeler wrote: Getting fail report emails is annoying and should be changed to be opt-in. Would that solve your problem? Oh, and yes. Once we stop spamming people, CPAN Testers then becomes the Consumer Reports model, not the police model. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 5, 2008, at 11:28, Andy Lester wrote: Getting fail report emails is annoying and should be changed to be opt-in. Would that solve your problem? Oh, and yes. Once we stop spamming people, CPAN Testers then becomes the Consumer Reports model, not the police model. Thank you. I think we're all in agreement on the benefit of an opt-in model, and that it would go a long way towards eliminating the complaints. At least, that's my reading of this thread. So what needs to happen to get this working sooner rather than later? Best, David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Friday 05 September 2008 11:23:07 David E. Wheeler wrote: And if you have to opt-in, I imagine that would solve the biggest complaint, yes? It's the unsolicited email reports that are annoying, right? They are annoying, but I'm not sure it's my biggest complaint. There's also the arbitrariness of the upload/debug/revise cycle of trying to please a black box full of testers. I'm not willing to say that this is primarily the fault of CPAN Testers, but it does expose a lot of cracks in the CPAN plumbing. It's a little bit like trying to have a discussion with someone who's upset but won't tell you why, and you have to guess and hope you don't make things worse before you get a useful answer. Well, you can upload a dev version to CPAN and the testing bots will test it, I believe. It'd be nice if there was a separate place to upload code to be tested before you actually released it. That'd be very handy indeed. Even being able to identify from a distribution which CPAN Testers platforms will even try to run tests would help. (Oh dear, this'll get all of those 5.005 boxes running my code.) I do like how CPANTS lists the original dozen or so Kwalitee metrics and their solutions on the individual distribution Kwalitee pages. -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
* chromatic [EMAIL PROTECTED] [2008-09-05 19:50]: If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. That’s CPANTS, the useful part of it anyway… except that the feedback cycle is even *much* slower there than with the Testers. CPANTS is really miscast as a service; the useful bits should really be extracted and made part of the `distcheck` targets of the various Perl distro toolchains. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
In article [EMAIL PROTECTED], chromatic [EMAIL PROTECTED] wrote: If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. That's what Module::Release does, and since the QA Hackathon in Oslo, it even tests under multiple perls. It can check kwalitee (or not), or anything else that you like.
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Friday 05 September 2008 16:52:05 brian d foy wrote: In article [EMAIL PROTECTED], chromatic If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. That's what Module::Release does, and since the QA Hackathon in Oslo, it even tests under multiple perls. It can check kwalitee (or not), or anything else that you like. Hm. What's your thought on turning that into something which a Module::Build or ExtUtils::MakeMaker plugin could run on ./Build distcheck or make distcheck? I'm happy to write the Module::Build plugin, if anyone else might find this idea useful. (I'd very happily explain to new CPAN contributors the value of running those commands against distributions before upload.) -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Fri, Sep 5, 2008 at 6:17 PM, chromatic [EMAIL PROTECTED] wrote: On Friday 05 September 2008 16:52:05 brian d foy wrote: In article [EMAIL PROTECTED], chromatic If I could see somehow that my distribution implicitly runs on Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL or Build.PL, or any of the other dozens of packaging quirks that can cause problems, I could fix them before uploading and before triggering a wave of testing. That's what Module::Release does, and since the QA Hackathon in Oslo, it even tests under multiple perls. It can check kwalitee (or not), or anything else that you like. Hm. What's your thought on turning that into something which a Module::Build or ExtUtils::MakeMaker plugin could run on ./Build distcheck or make distcheck? Module::Release already does that, too. You don't run it from make, you just type 'release' at the command line and it checks the `make test` and `make disttest`. There is a Module::Build plugin, but I haven't verified that it works for my latest versions yet (it probably doesn't). If someone wants to integrate it some other way, they can, but I have a different workflow that works a lot better for me. Module::Release tests a bunch of things, and if anything fails, it stops and doesn't release anything. The major bits are the distro tests, prereqs (I'm moving Test::Prereq out of my dists and into Module::Release), kwalitee, and source control status. Although you can look at the 'release' script in search.cpan.org, here's the output of a dry run to show you how it works (even more stuff happens for a real release after all this stuff passes)l: macbookpro_brian[686]$ release -t Testing with /usr/local/bin/perl5.10.0 Cleaning directory... no Makefile---skipping Recreating make file... done Running make... done Checking make test... all tests pass Checking prereqs... done Making dist... done Checking make disttest... all tests pass Testing with /usr/local/bin/perl Cleaning directory... done Recreating make file... done Running make... done Checking make test... all tests pass Checking prereqs... done Making dist... done Checking make disttest... all tests pass Testing for kwalitee Cleaning directory... done Recreating make file... done Running make... done Making dist... done Checking kwalitee... done Checking source repository Checking state of Git... Git up-to-date on branch 1 Cleaning directory... done -- brian d foy [EMAIL PROTECTED] http://www.pair.com/~comdog/
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
# from David Golden # on Wednesday 03 September 2008 14:09: if ... CPAN Testers is ... to help authors improve quality rather than ... to give users a guarantee about ... any given platform. ... high quality author -- ... tester has a broken or misconfigured toolchain The false positive ratio will approach 100%. __Repetition is desensitizing__ ... [more data will] improve confidence in the aggregate result. ... However, ... only need to hear it once. ... fail during PL or make ... valuable information to know when distributions can't build ... [or] ... don't pass tests. ... not just skip reporting. ... [but] false positives that provoke complaint are toolchain issues during PL or make/Build. ... no easy way to distinguish the phase ... subject of an email. I... partitions the FAIL space in a way that makes it easier for authors to focus on which ever part of the PL/make/test cycle... Yes. Important points. Excellent assessment. __What we can fix now and what we can't__ ... However, as long as the CPAN Testers system has individual testers emailing authors, there is little we can do to address the problem of repetition. This is true. For me, the mail is such a completely random data point (no PASS mail, not all of the FAIL mail, etc) that I just delete it. Once that is more or less reliable, we could restart email notifications from that central source if people felt that nagging is critical to improve quality. Personally, I'm coming around to the idea that it's not the right way to go culturally for the community. Send one mail to each new author telling them about the setup and how to get rss or configure it to mail them, etc. Telling people about something they don't know *once* is usually helpful. But with the per-tester direct mail, the recipient is powerless to stop it, and feeling powerless tends to make people angry. In the bigger scheme of things, I sort of think all new PAUSE authors should get a welcome e-basket with links, advice, etc, along with perhaps an optional (default?) monthly newsletter about new tools or something. But, the trend generally seems to be away from mail and towards web services. I'm in favor. Some of these proposal would be easier in CPAN Testers 2.0, which will provide reports as structured data instead of email text, but if exit 0 is a straw that is breaking the Perl camel's back now, then we can't ignore 1.0 to work on 2.0 as I'm not sure anyone will care anymore by the time it's done. What are the issues in that time? Code or non-code? What we can't do easily is get the testers community to upgrade to newer versions of the tools. That is still going to be a matter of announcements and proselytizing and so on. But I think we can make a good case for it, and if we can get the top 10 or so testers to upgrade across all their testing machines then I think we'll make a huge dent in the false positives that are undermining support for CPAN Testers as a tool for Perl software quality. Yes. Further, if you can detect new tools from old, you have enough information to filter the results. Hah! You know, you have their e-mail address, right? You can just send them nagging mail about how their tools are FAIL! ;-) So, seriously Testing is good. Knowing that weird platforms fail or pass is good. Knowing that old toolchains fail is even useful to some extent, but I think the significant variables are different for different users. So, what are these use cases? Let's pretend that I'm a real jerk of an author and I only care about whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky vendor patches) with a fully updated and properly configured toolchain and the clock set to within 0.65s of NTP time. It would then be useful for me to be able to mark (for all the world to see) which of the machines reporting were using supported configurations. That would both show *me* a valid fail and let my users know whether their configuration was supported. The user could still see passes and fails (hey, they could even perhaps login and set a preference for what to show.) And remember that the user is also often an author who is considering whether a dependency is a good choice for them or not. If they need to support perl 4 and like having wonky vendor perls and purposefully set their clock 5min fast and configure CPAN.pm for prefer_installer = rand, seeing those results are helpful -- but seeing my implied unsupported configuration flag is also helpful. Now, I'm not really a jerk, so I don't need the thing about the clock. I can probably live with false results from some redhat perl for a while too. But I'm going to pretend that we do not have a broken or impossible to upgrade toolchain until it comes true. My users do not have to share that delusion, but they do have to humor me. So, what do you show the author and/or users by default? That is possibly still rather sticky, but making
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 4, 2008 at 4:19 AM, Eric Wilhelm [EMAIL PROTECTED] wrote: Some of these proposal would be easier in CPAN Testers 2.0, which will provide reports as structured data instead of email text, but if exit 0 is a straw that is breaking the Perl camel's back now, then we can't ignore 1.0 to work on 2.0 as I'm not sure anyone will care anymore by the time it's done. What are the issues in that time? Code or non-code? mostly $job, @family, %life stuff Yes. Further, if you can detect new tools from old, you have enough information to filter the results. Hah! You know, you have their e-mail address, right? You can just send them nagging mail about how their tools are FAIL! ;-) I'd already come to the same realization, actually. So, what do you show the author and/or users by default? That is possibly still rather sticky, but making it adjustable (perhaps according to the maintainer by default) should allow reasonable people to be happy enough. That's the stuff that I'd probably defer to CPAN Testers 2.0. Once test reports are structured data, it will be much easier to filter on toolchain, environement, etc. Right now, someone would have to effectively screen scrape the reports -- with variations for CPANPLUS, CPAN::Reporter and their different formats over time. It could be done -- this is Perl after all -- but that's not where I'm interested in putting my energies. -- David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Two cents from someone who appreciates the hell out of the CPAN testing service and eagerly awaits new reports every time I release a new version of a module. However, from author's perspective, if a report is legitimate (and assuming they care), they really only need to hear it once. Having more and more testers sending the same FAIL report on platform X is overkill and gives yet more encouragement for authors to tune out. So the more successful CPAN Testers is in attracting new testers, the more duplicate FAIL reports authors are likely to receive, which makes them less likely to pay attention to them. Sorry, but paying attention is the author's job. A fail is something that should be fixed, period, regardless of the number of them. As mentioned elsewhere, the idea of author's receiving FAIL reports is outdated anyway: they should be pulling them via a RSS feed. First, we can lower our collective tolerance of false positives -- for example, stop telling authors to just ignore bogus reports if they don't like it and find ways to filter them. +1 Second, we can reclassify PL/make/Build fails to UNKNOWN. I don't like this: failure by any other name would smell just as bad. In other words, if an end user is not going to have a happy, functional module after typing install Foo::Bar at the CPAN prompt, this is a failure that should be noted as such and fixed by the author. Makefiles have a surprising amount of power and flexibility in this regard. However, as long as the CPAN Testers system has individual testers emailing authors, there is little we can do to address the problem of repetition. Yep. Use RSS or deal with the duplicates, I say. For those who read this to the end, thank you for your attention to what is surely becoming a tedious subject. Thanks for raising it. I honestly feel the problem is not with the testers or the testing service, but the authors. But perhaps I'm still grumpy from the slew of modules I've come across on CPAN lately that are popular yet obviously unmaintained, with bug reports, questions, and unapplied patches that linger in the RT queues for years. It would be nice if we had some sort of system that tracked and reported on that. -- Greg Sabino Mullane [EMAIL PROTECTED] End Point Corporation signature.asc Description: PGP signature
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 10:30 AM, Greg Sabino Mullane wrote: So the more successful CPAN Testers is in attracting new testers, the more duplicate FAIL reports authors are likely to receive, which makes them less likely to pay attention to them. Sorry, but paying attention is the author's job. A fail is something that should be fixed, period, regardless of the number of them. According to who? Who's to say what my job as an author is? -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thursday 04 September 2008 01:19:44 Eric Wilhelm wrote: Let's pretend that I'm a real jerk of an author and I only care about whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky vendor patches) with a fully updated and properly configured toolchain and the clock set to within 0.65s of NTP time. Oh great, yet another check I have to add to my Build.PL. What's the magic cantrip for THAT one? (Why yes, I *have* seen bugs related to time skew on network-mounted paths), -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thursday 04 September 2008 08:30:19 Greg Sabino Mullane wrote: Sorry, but paying attention is the author's job. A fail is something that should be fixed, period, regardless of the number of them. My job is editor, not programmer. Also novelist -- but again, not programmer. Certainly not CPAN programmer. I volunteer as a programmer, but your goals don't automatically become my goals because you want them to become my goals. They only become my goals if we reach some sort of contractual arrangement with mutual consideration, and then only for the scope of the contract. Paying attention is not my job. Releasing software I've written under a free and open license does not instill in me the obligation to jump whenever a user snaps his or her fingers. I may do so because I take the quality and utility of my software seriously, but do not mistake that for anything which may instill in you any sort of entitlement. That is an excellent way not to get what you want from me. I don't like this: failure by any other name would smell just as bad. In other words, if an end user is not going to have a happy, functional module after typing install Foo::Bar at the CPAN prompt, this is a failure that should be noted as such and fixed by the author. Then CPAN Testers reports should come with login instructions so that I can resurrect decade-old skills and perform system administration to fix broken installations and configurations -- oh, and as you say, a truly *modern* reporting system should publish these logins and passwords publicly in syndication feeds. (I do see a couple of problems with that idea, however.) I fail to understand the mechanism by which CPAN Testers has seemingly removed the ability of testers to report bugs to the correct places. For example, consider Shlomi's earlier message suggesting that a misconfigured CPAN configuration (caused by a bug in CPAN.pm) was the cause of FAIL reports for one of Shlomi's distributions. (I saw and responded to a similar report earlier this week.) Yes, there was a bug and yes, it's important to get it fixed. However, by what possible logic can you conclude that the appropriate way to get that bug fixed is to report it to people who, given all of the information detected automatically, *do not* maintain CPAN.pm? Oh, perhaps you think, it's easy for them to read the reports and diagnose the problem remotely on machines they have never seen before, did not configure, and cannot probe -- and it's so easy for them to file a bug in the right place! If you don't think that, precisely what *do* you think to produce such a bold assertion that it is Shlomi's job to install and reconfigure a new version of CPAN.pm for a CPAN Tester -- or for that matter, everyone with a misconfigured version of CPAN.pm which contains this bug? It's not my job to fix bugs in *my own* distributions. I do it because I care about quality and, contrary to what appears to be near-universal belief around here, I care that people can use my code. It is most assuredly not my job to fix bugs in distributions I've never maintained, nor report bugs in distributions I may not use, nor report bugs I haven't encountered and diagnosed myself. (The parallels to challenge-response email systems amuse me.) Thanks for raising it. I honestly feel the problem is not with the testers or the testing service, but the authors. But perhaps I'm still grumpy from the slew of modules I've come across on CPAN lately that are popular yet obviously unmaintained, with bug reports, questions, and unapplied patches that linger in the RT queues for years. It would be nice if we had some sort of system that tracked and reported on that. Besides rt.cpan.org? -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 04, 2008 at 10:09:20AM -0700, chromatic wrote: I fail to understand ... that much is obvious ... the mechanism by which CPAN Testers has seemingly removed the ability of testers to report bugs to the correct places. What a lovely straw man! Even nicer than the previous one. For example, consider Shlomi's earlier message suggesting that a misconfigured CPAN configuration (caused by a bug in CPAN.pm) was the cause of FAIL reports for one of Shlomi's distributions. (I saw and responded to a similar report earlier this week.) Change the record, please. This one's getting boring. Maybe I should start being equally loud and obnoxious about obviously stupid and broken things like the existence of UNIVERSAL-isa. It might give you some appreciation for how you're coming across here. -- David Cantrell | Enforcer, South London Linguistic Massive Perl: the only language that makes Welsh look acceptable
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Sorry, but paying attention is the author's job. A fail is something that should be fixed, period, regardless of the number of them. According to who? Who's to say what my job as an author is? Obviously I should be semantically careful: job and author are overloaded words. How about this: It's a general expectation among users of Perl that module maintainers are interested in maintaining their modules, and that said maintainers will try their best to remove any failing tests (that are under their power to do so). The parenthetical bit at the end is in response to the broken-CPAN straw man argument. Obviously (rare) things like that are out of control of the author, along with bugs in any other dependencies, OS utility, etc. I recognize that CPAN is a volunteer effort, but it does seem to me there is a implicit responsibility on the part of the author to maintain the module going forward, or to pass the baton to someone else. Call it a Best Practice, if you will. The end-user simply wants the module to work. Maintainers not paying attention, and the subsequent bitrot that is appearing on CPAN, is one of Perl's biggest problems at the moment. Careful attention and responsiveness to CPAN testers and to rt.cpan.org is the best cure for this. -- Greg Sabino Mullane [EMAIL PROTECTED] End Point Corporation signature.asc Description: PGP signature
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 12:50 PM, David Cantrell wrote: I fail to understand ... that much is obvious And here we have the core problem. chromatic, among others, have expressed frustration about CPAN Testers. The reaction has never been positive. Here, chromatic is insulted for simply saying I would like it if... -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 4, 2008 at 1:09 PM, chromatic [EMAIL PROTECTED] wrote: I fail to understand the mechanism by which CPAN Testers has seemingly removed the ability of testers to report bugs to the correct places. For example, I think it's a mistake to set this up as just an author-vs-tester zero-sum situation. I see this as a team game where (hopefully) we're all pretty much on the same side in that we want stuff to work. It shouldn't be any big deal to report a failure -- once -- to an author. That's just the normal bug-report cycle as an author might get from any human user. Author can look into it (if they care to), decide if it's a legitimate bug of theirs or if it's upstream. In some cases upstream is the toolchain. In other cases, it's dependencies. For example, RJBS noted how a CPAN Testers report helped him find a dependency issue: http://use.perl.org/~rjbs/journal/37336 The difference with toolchain-driven failures is that authors often can't really do anything about them directly. They can add workarounds, yes, but not all authors want to spend their time doing that. (Nor should we expect them too, really.) Usually, the easiest answer is for the end-user to upgrade their toolchain. That's why we need to partition the failure types, so that authors can distinguish between test failures -- which we presume they are likely to be interested in addressing -- instead of PL/make failures, which they may not be interested in addressing. It's not to say that one is less of a problem for end-users. Using UNKNOWN is just convenient at the moment because it exists already and is minimally used. That said, it should still be responsibility of testers to ensure they have a reasonably sane configuration that could potentially be successful at building and testing a distribution. It does very little good to have a broken CPAN that causes Build -j3 errors -- no Build.PL could ever succeed and so the fact that a Build.PL dist failed isn't telling us anything valuable about the distribution. -- David
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
I may do so because I take the quality and utility of my software seriously, but do not mistake that for anything which may instill in you any sort of entitlement. That is an excellent way not to get what you want from me. It's not an entitlement, it's a shared goal of making Perl better. If a maintainer is going to ignore test reports, perhaps its time to add a co-maintainer. Then CPAN Testers reports should come with login instructions so that I can resurrect decade-old skills and perform system administration to fix broken installations and configurations -- oh, and as you say, a truly *modern* reporting system should publish these logins and passwords publicly in syndication feeds. (I do see a couple of problems with that idea, however.) Besides the number of straw men starting to fill the room? However, by what possible logic can you conclude that the appropriate way to get that bug fixed is to report it to people who, given all of the information detected automatically, *do not* maintain CPAN.pm? That's not my argument at all. But the great majority of non-CPAN.pm-editing build errors can be fixed by the maintainer. Or at least routed around for testing purposes. Oh, perhaps you think, it's easy for them to read the reports and diagnose the problem remotely on machines they have never seen before, did not configure, and cannot probe -- and it's so easy for them to file a bug in the right place! I don't think this is easy at all. But it's also not quite the burden you make it appear. All the testers I've contacted about helping me fix test failures (build or otherwise) have been friendly and responsive. and unapplied patches that linger in the RT queues for years. It would be nice if we had some sort of system that tracked and reported on that. Besides rt.cpan.org? Yes, something that indicates the age and number of open bugs for a module, the age of any unapplied patches, and perhaps some other metrics to indicate maintainedness. Cross referencing that with popularity and dependency chains would be a great triage system to start whipping CPAN into shape. -- Greg Sabino Mullane [EMAIL PROTECTED] End Point Corporation signature.asc Description: PGP signature
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thursday 04 September 2008 11:30:51 David Golden wrote: It shouldn't be any big deal to report a failure -- once -- to an author. That's just the normal bug-report cycle as an author might get from any human user. Author can look into it (if they care to), decide if it's a legitimate bug of theirs or if it's upstream. In some cases upstream is the toolchain. In other cases, it's dependencies. I can live with occasionally having to triage tricky reports which may need resolution elsewhere, provided that: * CPAN Testers clients filter out the most common cases of toolchain misconfiguration (in progress, probably will require ongoing maintenance) * Duplicate reports get filtered out (in planning) * CPAN Testers appear to have done some triaging of failures on their own (at least reading the report and deciding if it's appropriate to send to the author before sending it) These would make the process more worthwhile to me. I only speak for myself here, of course, and you're well within your rights to ignore my suggestions or desires here. That said, it should still be responsibility of testers to ensure they have a reasonably sane configuration that could potentially be successful at building and testing a distribution. It does very little good to have a broken CPAN that causes Build -j3 errors -- no Build.PL could ever succeed and so the fact that a Build.PL dist failed isn't telling us anything valuable about the distribution. Yes! That's the philosophy I want applied to all sorts of tests. Does this test tell me anything valuable? Does this test tell me anything actionable? Is that information worth the cost of the test? (Don't worry; this is not a problem specific to CPAN Testers. I see it in plenty of test suites, all the time.) -- c
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Hi Andy - On Thu, Sep 4, 2008 at 1:53 PM, Andy Lester [EMAIL PROTECTED] wrote: Why should I release my software on CPAN if part of the price of entry is being spammed and told what I should be doing? Although the remark about CPAN authors' jobs was worded less than optimally, part of the context of it was that these reports should not get emailed directly to the authors from the testers. That sounds like a point of very widespread agreement. It allows interested authors and interested users to optionally check on test results. It also sounds like there is work being done to cut down on some of the toolchain problems that get reported as failures. Do these two things help make the CPAN Testers stuff more useful or at least less annoying for you? -Andy
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Sep 4, 2008, at 2:11 PM, Andrew Moore wrote: Do these two things help make the CPAN Testers stuff more useful or at least less annoying for you? The only thing that will make CPAN Testers less annoying at this point is if I am ASKED WHAT I WANT, instead of being told Here's what we're doing and dammit, you should like it! It is a problem of attitude. Who is serving who? Who is the customer? xoxo, Andy -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 4, 2008 at 2:21 PM, Andy Lester [EMAIL PROTECTED] wrote: The only thing that will make CPAN Testers less annoying at this point is if I am ASKED WHAT I WANT, instead of being told Here's what we're doing and dammit, you should like it! You're right, Andy. I was being sort of vague there. What do you want? Do you see something (that hasn't already been discussed here in the last two days) that the people writing this code could do to make it more useful for you? Please note: I have never contributed anything to the CPAN testing project other than a few reports. It looks to me like there's a bit of a communication gap here that I'm hoping to fill for just a moment. There's a lot of people talking past each other instead of with each other. The contributors appear to be interested in making a useful tool, though, and it sounds like we're making progress. -Andy
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
Citeren Andy Lester [EMAIL PROTECTED]: On Sep 4, 2008, at 1:33 PM, Greg Sabino Mullane wrote: It's not an entitlement, it's a shared goal of making Perl better. If a maintainer is going to ignore test reports, perhaps its time to add a co-maintainer. Yes, something that indicates the age and number of open bugs for a module, the age of any unapplied patches, and perhaps some other metrics to indicate maintainedness. Cross referencing that with popularity and dependency chains would be a great triage system to start whipping CPAN into shape. Maybe what's so frustrating to me, and perhaps to chromatic, and whoever else ignores CPAN Testers but doesn't discuss it, is that we're being fed things that we should be thankful for and goddammit why aren't we appreciative??!? You shouldn't be thankful or appreciative. (IMHO) The community should be (and is) thankful that you chose to put your code out there for everyone to use and improve. Automated testing is easy and (I believe) isn't that time consuming. Writing code, solving bugs, adding features and making the code more available (platform/configuration indipendent) isn't easy and certainly can consume lots of time. Thanks should go to the person that spends/spended the most time on this. Meaning the author and contributors. Here are the things that we have determined are quality. Here are test reports reporting on failures for these things that we care about you caring about. Maybe you should add a co-maintainer. Your responsibility as an author is to... It's obvious that the main idea behind these messages is to improve the quality of the distributions on CPAN. What the messages are ignoring is that it isn't (IMHO) the task/job of the author. In a perfect world each FAIL report would come with a patch attached that the author can chose to apply. Unfortunally we are not living in a perfect world... Is a FAIL under configuration XYZ a problem? To some (real) users it certainly can be. But they have three choiches: - Find the source of the problem and fix it themself (and contribute a patch) - Contact the author asking for help - Look for another solution CPAN Testers is entirely based on the concept of *unsolicited advice* in the name of helping the author. In essence it is (IMHO) about helping the users of the module and not the author. The goal is to make the module more available. If the author of the module choses to do so then that is great! If the author does not then that is perfectly fine and then the FAILs still serve a purpose: - someone else might care a lot about your module and noticing the FAIL reports and starts patching - the users are (in theory) informed that the module is not expected to work under configuration XYZ. (Obviously it would be best that the author in that case isn't spammed with FAIL reports but I belive work for that is in progress.) From a beautiful article I've reproduced at http://xoa.petdance.com/Unsolicited_advice == snip == Life's little helpers reason that the first step toward improvement is the realization that things need to be improved. That is why they feel justified in approaching you when you are perfectly content in order to point out that everything you do, eat, and love is a dreadful mistake. Because they themselves are so full of good wishes for the rest of humanity, they do not expect their beneficiaries to be petty. They figure that upon being told how you have mismanaged your life, you will be grateful for the offer of assistance and reassured that others are watching out for you. It stands to reason that one who obviously does not know what is best for himself would be relieved to find that others are willing to take on that responsibility. After all, they don't just stop after telling you that is wrong, but always go on to explain in detail how you can do things the way they do them. In other words, the right way. == snip == And then, when we say OK, I'm not interested in stopping smoking, losing weight, or checking for Perl version 5.6.1, we're told how full of shit we are. Which is a pity. Given that you already did a great job by making your code available to everyone. Why should I release my software on CPAN if part of the price of entry is being spammed and told what I should be doing? People will always complain. It's a lot easier to complain that the author isn't doing their 'job' 'correctly' then helping the author out by sending patches. What you shouldn't forget is you are hearing much less from the many many users that are using your software and are very grateful for and happy with it. Kind regards, Bram
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 04, 2008 at 06:50:37PM +0100, David Cantrell wrote: On Thu, Sep 04, 2008 at 10:09:20AM -0700, chromatic wrote: I fail to understand ... that much is obvious [etc] My apologies chromatic, I shouldn't have lost my temper and said that. -- David Cantrell | London Perl Mongers Deputy Chief Heretic Planckton: n, the smallest possible living thing
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Thu, Sep 04, 2008 at 02:21:23PM -0500, Andy Lester wrote: On Sep 4, 2008, at 2:11 PM, Andrew Moore wrote: Do these two things help make the CPAN Testers stuff more useful or at least less annoying for you? The only thing that will make CPAN Testers less annoying at this point is if I am ASKED WHAT I WANT, instead of being told Here's what we're doing and dammit, you should like it! You've already made it perfectly clear to me that you have no interest in receiving reports at present. That's why I stopped sending them to you. I'll do the same for anyone. You don't have to like it. What you do have to accept, however, is that like just about everything else perlish the systems in place grew in an ad-hoc fashion because us perl programmers have an annoying tendency to just fucking do it. We see a problem, we solve it. We see something that we think needs doing, we do it. And then people build further ad-hoccery on top of that, and so on. That's why we have the regrettable situation *currently* that you are asked to opt out instead of opting in. Sorry, but that's the way it is at the moment. Want another example of ad-hoc JFDI being bad? Schwern can break CPAN. He's done it at least once recently. It got fixed because the people affected told him. It would have been better if he'd passed his changes to a release manager who would make sure they were all thoroughly tested against the entire CPAN before release. You may have noticed that when the CPAN-testers screw up *and we're told about it* things get fixed too. If I send a bogus report, I want to know, and then if it's within my power I'll fix it and if it's not I'll pass it up the chain to the right person. What I'm not willing to do, however, is to manually check every report and ensure perfection that way. Why? Because it takes too long, and I have a job and a life. And anyway, I'd still make mistakes - and even if I don't make mistakes, people will still think I have. Everyone makes mistakes when they're doing a boring job, doubly so without the prospect of reward. So anyone who insists that I read every report I send to them will just get no reports from me. Again, all you have to do to stop me sending you reports is *tell me to stop sending you reports*. -- David Cantrell | Nth greatest programmer in the world comparative and superlative explained: Huhn worse, worser, worsest, worsted, wasted
Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)
On Wed, Sep 3, 2008 at 4:09 PM, David Golden [EMAIL PROTECTED] wrote: I'm interested in feedback on these ideas -- on list or off. In particular, I'm now convinced that the success of CPAN Testers now prompts the need to move PL/make fails to UNKNOWN and to discontinue copying authors by individual testers. I'm open to counter-arguments, but they'll need to convince me of a better long-run solution to the problems I identified. David, These two steps sound like they would go a long way toward making this stuff less annoying for some, and just as useful for the rest. I also think It's a good stop-gap measure to quiet the crowd while improvements are coming. Thanks! -Andy
