Re: What do you want? (was Re: The relation between CPAN Testers and quality)
* Andy Lester [EMAIL PROTECTED] [2008-09-05 06:45]: I want nothing in my inbox that I have not explicitly requested. I want to choose how I get reports, if at all, and at what frequency. I want aggregation of reports, so that when I send out a module with a missing dependency in the Makefile.PL, I don't get a dozen failures in a day. (Related, but not a want of mine, it could aggregate by platform so I could see if I had patterns of failure in my code). I want to be able to sign up for some of this, some of that, on some of those platforms. These are currently difficult as a matter of architecture, which is “every tester sends mail directly to every author.” The design overhaul will centralise the gathering and issuing of reports, so all of these things will become possible in the medium term. You are not the only one to ask for them, FWIW. I want to select what kwalitee benchmarks I choose my code to be verified under, so that I can proudly say My modules meet these criteria across these platforms. I want a couple dozen checkboxes of things that could be checked where I say All my modules had better match test X, Y and Z, and these specific modules had also better past A, B and C, too. I want easily selected Kwalitee settings which group together options. Slacker level means you pass these 10 tests, and Lifeguard level means you are Slacker + these other 15 tests, and Stringent level means something else, all the way up to Super Duper Batshit Crazy Anal Perfection level. It seems that you are confusing the CPAN Testers with the CPAN Testing Service (CPANTS), Domm’s pet project. The only relation between the two is their confusingly similar naming. CPANTS tries to lint-check your distribution and code without running any of it; the CPAN Testers download your releases, install any prereqs and then run your test suite. The goals and designs of the two projects as well as their participants are entirely different. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Fri, Sep 5, 2008 at 12:42 AM, Andy Lester [EMAIL PROTECTED] wrote: I want nothing in my inbox that I have not explicitly requested. I want to choose how I get reports, if at all, and at what frequency. I'm going to take the first steps towards this over the weekend by deprecating author CC's in the code and encouraging top testers to upgrade their tools. That has the disadvantage of stopping FAIL reports for everyone, including those who want them so in the mean time, I or one of the other CPAN Testers plan to work up a centralized notification that either limits a FAIL email to one per dist/arch/perl-version tuple or else puts all reports into a single daily digest. That isn't entirely nothing in your inbox, but it should be a dramatic reduction while we figure out where and how to let authors set preferences of this sort. I would imagine that in the meantime we will have an exclusions list for authors to skip, but it will have to be manually maintained until we set up a system to automate preference management. I want aggregation of reports, so that when I send out a module with a missing dependency in the Makefile.PL, I don't get a dozen failures in a day. (Related, but not a want of mine, it could aggregate by platform so I could see if I had patterns of failure in my code). Have you seen http://matrix.cpantesters.org ? That's the answer to the related part of your question. I want to be able to sign up for some of this, some of that, on some of those platforms. That will probably be possible once we figure out how to let authors specify preferences of that type. I want CPAN Testers to do what I can not easily do, which is test my code on other platforms on other versions of Perl. Well, this, at least, we're doing today. I do NOT want CPAN Testers to do what I could easily do if I wanted, but do not, which is run tests that I don't care about. Fortunately, we're not doing this. (As someone mentioned, CPAN Testers is not CPANTS.) Unless you count testing for prerequisites which we're doing only because they're part of the PL/make/test cycle. I want the Ruby guys go holy shit, I wish we had something like that. +1 -- David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
In article [EMAIL PROTECTED], David Golden [EMAIL PROTECTED] wrote: On Fri, Sep 5, 2008 at 12:42 AM, Andy Lester [EMAIL PROTECTED] wrote: I want nothing in my inbox that I have not explicitly requested. I want to choose how I get reports, if at all, and at what frequency. I'm going to take the first steps towards this over the weekend by deprecating author CC's in the code and encouraging top testers to upgrade their tools. I remember some discussion about setting some thing to opt-in or -out of those. I'd hate to lose those in my email because other people don't want to filter their mail.
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 4, 2008, at 21:42, Andy Lester wrote: I want nothing in my inbox that I have not explicitly requested. Yes, for email reports, it'd be nice to subscribe to a list of your own reports -- and to be able to request which reports you want (fail only, non-pass, all, etc.). I want to choose how I get reports, if at all, and at what frequency. I want aggregation of reports, so that when I send out a module with a missing dependency in the Makefile.PL, I don't get a dozen failures in a day. (Related, but not a want of mine, it could aggregate by platform so I could see if I had patterns of failure in my code). This makes sense for email reports, but if you use the RSS feed, it's not so important. As CPAN testers moves away from using email to submit reports, this should get a lot better. I can't wait, frankly. I want to be able to sign up for some of this, some of that, on some of those platforms. Schedule that for CPAN Testers 3.0. ;-) I want suggestions, not mandates, in how I might improve my code. I want explanations on my CPAN Testers dashboard that explains why I would be interested in having such-and-such an option checked on my distributions. See how the Perl::Critic policies have explanations of the problem, and why it can be a problem, in the docs for the code. I want CPAN Testers to be as flexible as Perl::Critic, and even easier to do that flexing. For a volunteer effort, this could be quite tricky. I want the understanding that not everyone shares the same coding ideals. Done. Boy, that was easy! ;-P I want to select what kwalitee benchmarks I choose my code to be verified under, so that I can proudly say My modules meet these criteria across these platforms. I want a couple dozen checkboxes of things that could be checked where I say All my modules had better match test X, Y and Z, and these specific modules had also better past A, B and C, too. That's CPANTS, not CPAN Testers. I want easily selected Kwalitee settings which group together options. Slacker level means you pass these 10 tests, and Lifeguard level means you are Slacker + these other 15 tests, and Stringent level means something else, all the way up to Super Duper Batshit Crazy Anal Perfection level. Also more like CPANTS. CPAN Testers is all about two things: Does the module build and do all the tests pass. Nothing more. For all those metrics, you'd have to include them in your tests, methinks. I want CPAN Testers to do what I can not easily do, which is test my code on other platforms on other versions of Perl. That already happens. My modules are a lot better for it (mainly thanks to David Golden and Slaven Rezic. I do NOT want CPAN Testers to do what I could easily do if I wanted, but do not, which is run tests that I don't care about. Don't ship those tests, then. I want CPAN Testers to be a service where people say Hey, have you seen CPAN Testers? You've got to check it out, it will help out your code so much and then they tell their friends and they tell their friends, and passing a certain batter of CPAN Testers tests consistently is a badge of honor. I want the Ruby guys go holy shit, I wish we had something like that. Heh. Best, David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
I'd hate to lose those in my email because other people don't want to filter their mail. I'd hate to get spammed because other people don't want to sign up to receive them. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 5, 2008, at 09:10, Andy Lester wrote: I'd hate to lose those in my email because other people don't want to filter their mail. I'd hate to get spammed because other people don't want to sign up to receive them. I think that adding changing things so that authors opt-in to getting reports is something that's generally been agreed on, though it might have to wait until the Web service is done. In the meantime, David Golden, at least, has tweaked his reporting toolchain so that it no longer submits reports to you, Andy. Does that work for you? Best, David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
In article [EMAIL PROTECTED], Andy Lester [EMAIL PROTECTED] wrote: I'd hate to lose those in my email because other people don't want to filter their mail. I'd hate to get spammed because other people don't want to sign up to receive them. You keep saying spam, but that's not the right term. You're being an ass characterizing it like that. Right now there is not a way to opt in, but there is a way to ignore it if you don't want to see it. I'll sign up for it when there is a way to do it. It's not a matter of me not wanting to do something, but jsut dealing effectively with the current situation. So, Andy, put up or shut up. Send in the patches.
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 5, 2008, at 12:46 PM, brian d foy wrote: You keep saying spam, but that's not the right term. You're being an ass characterizing it like that. I knew before even opening this mail that it would contain a personal attack. Why is it necessary to insult people who have different opinions? -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 5, 2008, at 10:57, chromatic wrote: Full credit (and many thanks) to David Golden and others who are moving away from this model, but if I'm an ass for saying You know, that has a lot in common with spam and CPAN-related services with good intentions should carefully consider the effects of their actions on their constituents, then I'm proudly an ass as well. I think that such diplomatic descriptions are useful, and don't make you an ass. But simply calling it spam is not very useful. There is an underlying reason one says its spam or spam-like, and that's the important thing. Best, David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Fri, Sep 5, 2008 at 12:14 PM, David E. Wheeler [EMAIL PROTECTED] wrote: On Sep 5, 2008, at 09:10, Andy Lester wrote: I'd hate to lose those in my email because other people don't want to filter their mail. I'd hate to get spammed because other people don't want to sign up to receive them. I think that adding changing things so that authors opt-in to getting reports is something that's generally been agreed on, though it might have to wait until the Web service is done. In the meantime, David Golden, at least, has tweaked his reporting toolchain so that it no longer submits reports to you, Andy. Clarification -- David *Cantrell* has tweaked his tools so it doesn't submit to Andy already. Too many Davids in this discussion apparently. I will be changing Test::Reporter to stop all author CC'ing which will take effect when/if we convince existing testers to upgrade. I have vague plans to create a nag bot to detect testers using old tools and nag them to upgrade. I'll post about the near-term replacement notification plan in a separate email. -- David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 5, 2008, at 1:36 PM, David Golden wrote: I will be changing Test::Reporter to stop all author CC'ing which will take effect when/if we convince existing testers to upgrade. Thank you, sir. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Fri, Sep 5, 2008 at 2:03 PM, Eirik Berg Hanssen [EMAIL PROTECTED] wrote: You keep saying spam, but that's not the right term. You're being an ass characterizing it like that. Yeah, it's not the right term. While it meets the other criteria to qualify as spam by the classical definition, it would also have to be bulk. Which this is not. For what it's worth, *I* consider it not quite spam, but not really not far from it and don't object to people using that term. The more successful CPAN Testers is at getting more testers and more platforms tested, the more it becomes bulk. For example, in my stress test of CPAN::Reporter::Smoker, when I submitted more than 138,000 in a single month, I made a point to turn off author CCing, because I didn't think that sending 20,000 fail reports was a particularly friendly thing to do. -- David
Re: What do you want? (was Re: The relation between CPAN Testers and quality)
On Sep 4, 2008, at 2:27 PM, David Golden wrote: I'm not being snide. I've heard what you don't want. I hope that you see that there is interest in making things better. In no particular order: I want nothing in my inbox that I have not explicitly requested. I want to choose how I get reports, if at all, and at what frequency. I want aggregation of reports, so that when I send out a module with a missing dependency in the Makefile.PL, I don't get a dozen failures in a day. (Related, but not a want of mine, it could aggregate by platform so I could see if I had patterns of failure in my code). I want to be able to sign up for some of this, some of that, on some of those platforms. I want suggestions, not mandates, in how I might improve my code. I want explanations on my CPAN Testers dashboard that explains why I would be interested in having such-and-such an option checked on my distributions. See how the Perl::Critic policies have explanations of the problem, and why it can be a problem, in the docs for the code. I want CPAN Testers to be as flexible as Perl::Critic, and even easier to do that flexing. I want the understanding that not everyone shares the same coding ideals. I want to select what kwalitee benchmarks I choose my code to be verified under, so that I can proudly say My modules meet these criteria across these platforms. I want a couple dozen checkboxes of things that could be checked where I say All my modules had better match test X, Y and Z, and these specific modules had also better past A, B and C, too. I want easily selected Kwalitee settings which group together options. Slacker level means you pass these 10 tests, and Lifeguard level means you are Slacker + these other 15 tests, and Stringent level means something else, all the way up to Super Duper Batshit Crazy Anal Perfection level. I want CPAN Testers to do what I can not easily do, which is test my code on other platforms on other versions of Perl. I do NOT want CPAN Testers to do what I could easily do if I wanted, but do not, which is run tests that I don't care about. I want CPAN Testers to be a service where people say Hey, have you seen CPAN Testers? You've got to check it out, it will help out your code so much and then they tell their friends and they tell their friends, and passing a certain batter of CPAN Testers tests consistently is a badge of honor. I want the Ruby guys go holy shit, I wish we had something like that. xoxo, Andy -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
