Re: New kwalitee test, has_changes
Collecting any sort of coverage data is a complete bitch. Let me just say right now that doing it across _all_ of CPAN is flat out impossible. It's impossible. Is it possible to use PPI to count the number of tests in a module's test suite? More than 5 tests would probably mean the author added one of his own. Of course the problem here how to deal with tests that don't use Test:: modules. So maybe this is also impossible. Maybe another approach is to take the module's entire test suite and remove from it any code that looks like it came from pod.t, pod-coverage.t, 00.load.t or any other stock test boilerplate. If there's something left over, then the module probably has an actual test suite. Michael --- Michael Graham [EMAIL PROTECTED]
Re: New kwalitee test, has_changes
It's impossible. Quite. I believe the only way is for the author to do the Devel::Cover dance and forward the results. It also distributes the workload out to where it should be done. Since it's an optional step that has no direct bearing on the functionality of the module, it's a sign that the author takes care. In fact, uploading any coverage statistics would already be a sign of quality. No, it isn't. It's a sign of beaurocracy. As would be any other metric that has 100% failure for all CPAN modules when it is first added, and then expects the rest of the world to change to match it's view of quality. We already have two of these rediculous fail-by-default metrics (has_pod_tests and has_pod_coverage_tests or whatever it's called). Kwalitee should measure reality, not set measurements that require reality to be changed to match. Adam K
Re: New kwalitee test, has_changes
Paul Johnson wrote: On Fri, Sep 23, 2005 at 12:06:43PM +0200, Thomas Klausner wrote: Hi! On Fri, Sep 23, 2005 at 12:54:42PM +1000, Adam Kennedy wrote: Collecting any sort of coverage data is a complete bitch. Let me just say right now that doing it across _all_ of CPAN is flat out impossible. It's impossible. And remember folks, only perl can parse Perl. Well, at CODE anyway. And coverage is about code, not documents. I completly agree. However, the problem is only slightly harder than testing all of CPAN. Now, if somebody sets up a system to collect coverage data thats generated at various decentralised machines and provides a nice interface to the results, CPANTS might be able to use coverage statistics as a metric. I really doubt it. If you want to prove this can be done, go find us a percentage coverage in CPAN Testers. That's your limit for generating coverage. And it introduces the additional problem of scoring for non-universal coverage. At the moment we have universal coverage of all metrics for all modules (a way of saying yes or no). PPI will let us take that further as it has (for all intents and purposes) universal coverage as well. However, what do you do when you only have 90% coverage? How do you handle the math? Adam K
Re: New kwalitee test, has_changes
Michael Graham wrote: As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80%, and another for 100%, and how absolutely impossible it would be to set out to establish these points for all the modules on CPAN. But it would be Good. I think a point for = 80% would be okay (for some definition of 80%). But I think a more useful measure of kwalitee would be a 20%-30% coverage test. Something like that sounds much more reasonable than a high number. Of course, if you've seen the first third of the PPI talk you realise we still have all the problems of having to use perl itself... #!/usr/bin/perl use Test::More 'no_plan'; use Suitcase::Nuke trigger_in_seconds = 1; pass(Looks good); oops, there goes the neighbourhood. Collecting any sort of coverage data is a complete bitch. Let me just say right now that doing it across _all_ of CPAN is flat out impossible. It's impossible. Adam K
Re: New kwalitee test, has_changes
Hi!
On Fri, Sep 23, 2005 at 12:54:42PM +1000, Adam Kennedy wrote:
Collecting any sort of coverage data is a complete bitch. Let me just
say right now that doing it across _all_ of CPAN is flat out impossible.
It's impossible.
I completly agree.
Now, if somebody sets up a system to collect coverage data thats generated
at various decentralised machines and provides a nice interface to the
results, CPANTS might be able to use coverage statistics as a metric.
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$-gprint$_.$/}
Re: New kwalitee test, has_changes
Adam Kennedy wrote: Michael Graham wrote: [...] But I think a more useful measure of kwalitee would be a 20%-30% coverage test. Something like that sounds much more reasonable than a high number. Of course, if you've seen the first third of the PPI talk you realise we still have all the problems of having to use perl itself... #!/usr/bin/perl use Test::More 'no_plan'; use Suitcase::Nuke trigger_in_seconds = 1; pass(Looks good); oops, there goes the neighbourhood. Collecting any sort of coverage data is a complete bitch. Let me just say right now that doing it across _all_ of CPAN is flat out impossible. It's impossible. Quite. I believe the only way is for the author to do the Devel::Cover dance and forward the results. It also distributes the workload out to where it should be done. Since it's an optional step that has no direct bearing on the functionality of the module, it's a sign that the author takes care. In fact, uploading any coverage statistics would already be a sign of quality. David -- It's overkill of course, but you can never have too much overkill.
Re: New kwalitee test, has_changes
On Wed, Sep 21, 2005 at 10:28:33PM -0400, James E Keenan wrote: David Landgren wrote: demerphq wrote: You miss my point. Whether the code be cross-platform or cross-version, you need to aggregate the coverage results from all the environments your code is designed to run on. How is this done? By specifying more than one database when you run the cover program. How you get the databases to be visible is currently your problem. cp, nfs, smbmount, scp, mail - your choice. The databases will be merged and a combined report produced. On Wed, Sep 21, 2005 at 04:26:27PM +0200, demerphq wrote: And, it doesnt help that something about DC breaks the defined operator when dealing with overloaded objects. (yeah, he did say the code was alpha quality :-) Bug reports, especially those containing small, self contained test cases, go a long way to helping such problems get solved ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: New kwalitee test, has_changes
On 9/22/05, Paul Johnson [EMAIL PROTECTED] wrote:
On Wed, Sep 21, 2005 at 04:26:27PM +0200, demerphq wrote:
And, it doesnt help that something about DC breaks the defined
operator when dealing with overloaded objects. (yeah, he did say the
code was alpha quality :-)
Bug reports, especially those containing small, self contained test
cases, go a long way to helping such problems get solved ;-)
Unfortunately this bug is in the crazy netherworld of perl weirdness,
and im not even sure if I know where to start.
If you want to see try to run Data::Dump::Streamer 1.14 under DC. It
will die horribly in overload.t, yet when run without DC it is fine.
overload.t is essentially a test to make sure that DDS never triggers
an overloaded method when dumping an object, so every single
overloaded operator is overloaded to Carp. The latter behaviour is the
first problem that DC triggers, as Carp.pm has code like this in it:
sub format_arg {
my $arg = shift;
if (not defined($arg)) {
$arg = 'undef';
}
elsif (ref($arg)) {
$arg = defined($overload::VERSION) ? overload::StrVal($arg) : $arg;
}
That first defined triggers the overloaded method when run under DC,
but doesnt when not. That in itself triggers infinite recursion and
the program segfaults.
Once the Carp code is patched to be
sub format_arg {
my $arg = shift;
if (ref($arg)) {
$arg = defined($overload::VERSION) ? overload::StrVal($arg) : $arg;
}elsif (not defined($arg)) {
$arg = 'undef';
}
the infinite recursion goes away, but you start seeing the same error
throughout DDS. By eliminating all calls to defined() on possibly
overloaded vars the errors go away. (Unfortunately i havent released
1.15 yet so you can see the changes made to 1.14 to make it pass DC
testing)
Conclusion: DC interferes with the behaviour of the defined function
and somehow triggers overloaded methods.
Unfortunately DDS is a pretty hairy module that works on a pretty
hairy domain (accurately dumping perl data structures is not at all
straight forward) so its hard to say what is responsible.
Ill do my best to find a minimal example and to prepare patches for
Carp.pm so it doesnt have this problem in the future.
Yves
--
perl -Mre=debug -e /just|another|perl|hacker/
Re: New kwalitee test, has_changes
As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80%, and another for 100%, and how absolutely impossible it would be to set out to establish these points for all the modules on CPAN. But it would be Good. I think a point for = 80% would be okay (for some definition of 80%). But I think a more useful measure of kwalitee would be a 20%-30% coverage test. Right now 'has_tests' is satisfied by including the default tests that module-starter provides (00.load.t, and the pod tests). There are a lot of modules that have nothing beyond the default tests and yet they get their 'has_tests' point. Passing a low coverage bar would at least indicate that the author wrote *some* tests. If there's an easier way of finding out if there are actual tests, then that would be fine too. I think there's a big difference in kwalitee between a module that has only the default tests and a module with a hand-crafted test suite. One of the first things I do when checking out a new module is to check if there are more than three files in the t/ directory. Michael --- Michael Graham [EMAIL PROTECTED]
Re: New kwalitee test, has_changes
demerphq wrote: On 9/15/05, David Landgren [EMAIL PROTECTED] wrote: As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80% ... I have to wonder about how you handle modules that have code that is Perl version dependent or OS dependent. Many non-trivial modules end up having to work around one such incompatibility or another, and therefore can't get full coverage on a single perl/OS combination. You could argue that such modules should have their platform/version specific bits in seperate modules, like what File::Spec does. I'd not support that argument though - it would make stuff like ... warn(Windows isn't supported\n) if($^O =~ /win32/i); impractical. -- David Cantrell
Re: New kwalitee test, has_changes
David Cantrell wrote: demerphq wrote: On 9/15/05, David Landgren [EMAIL PROTECTED] wrote: As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80% ... I have to wonder about how you handle modules that have code that is Perl version dependent or OS dependent. Many non-trivial modules end up having to work around one such incompatibility or another, and therefore can't get full coverage on a single perl/OS combination. You could argue that such modules should have their platform/version specific bits in seperate modules, like what File::Spec does. I'd not support that argument though - it would make stuff like ... warn(Windows isn't supported\n) if($^O =~ /win32/i); impractical. If a module has extensive platform-dependent codepaths, it is impossible to achieve full coverage in a single run. One would have to aggregate the coverage databases from separate D::C runs from, for example, Unix, VMS and Win32. This is probably something only the author can do, with perhaps a willing person who can forward the results from platforms the author does not have access to. If an author went to such troubles, she would be deserving of a 48pt bold, orange Kwalitee point. I know I had my eyes opened by Devel::Cover. I thought I had pretty good coverage in Regexp::Assemble. In fact I had about 60%. I lifted it up to 100% statement coverage (some branching and conditional paths are never taken, but they are normal, for example C$foo or return 0). This toook about 15 hours of work over a week or so. I caught a couple of minor bugs and one rather big one in the process. To me, this is a mark of Quality. It would be good to have it as a Kwalitee metric, but I see no easy way. The simplest way I can see would be to have a META.yml key that contains a URI to the HTML D::C report. I would rule out adding a cover/ subdirectory to the distribution due to the impact it would have on CPAN repositories. David -- It's overkill of course, but you can never have too much overkill.
Re: New kwalitee test, has_changes
Hi!
On Wed, Sep 21, 2005 at 11:58:36AM +0200, David Landgren wrote:
To me, this is a mark of Quality. It would be good to have it as a
Kwalitee metric, but I see no easy way. The simplest way I can see would
be to have a META.yml key that contains a URI to the HTML D::C report. I
would rule out adding a cover/ subdirectory to the distribution due to
the impact it would have on CPAN repositories.
CPANTS will never use Devel::Cover directly, because CPANTS doesn't run the
code it's looking at.
I think somebody once suggested a central server/service to collect
Devel::Cover statistics. Maybe slightly parallel to how testers work, but
I'm not sure and haven't got any time ATM to ask google..
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$-gprint$_.$/}
Re: New kwalitee test, has_changes
Selon Thomas Klausner [EMAIL PROTECTED]: Hi! On Wed, Sep 21, 2005 at 11:58:36AM +0200, David Landgren wrote: To me, this is a mark of Quality. It would be good to have it as a Kwalitee metric, but I see no easy way. The simplest way I can see would be to have a META.yml key that contains a URI to the HTML D::C report. I would rule out adding a cover/ subdirectory to the distribution due to the impact it would have on CPAN repositories. CPANTS will never use Devel::Cover directly, because CPANTS doesn't run the code it's looking at. I think somebody once suggested a central server/service to collect Devel::Cover statistics. Maybe slightly parallel to how testers work, but I'm not sure and haven't got any time ATM to ask google.. I proposed a similar idea using the CPAN Testers infrastructure to collect the statistics. The idea is to just add the cover information in the mail sent by the smokers (which would have the corresponding option enabled). Currently, CPANPLUS and CPAN::YACSmoke are running the test suite and sending a PASS report if it successes, or a FAIL report with details if it fails. The idea is to re-execute the test suite with Devel::Cover once it already succeeded, pack up the cover information and then send the PASS report with the coverage information. Or it can be sent in another email. Then CPAN::WWW::Testers could be extended to gather the statistics and present them. I didn't provided the code yet because I got distracted by other modules and other hacks I was trying to add to CPANPLUS. Something important is to check whether it is possible to send coverage information without taking too much space and bandwith. A quick test shows that the Storable files used by Devel::Cover seem to compress quite well with gzip, so an option may be to just send it this way. -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.
Re: New kwalitee test, has_changes
On 9/21/05, David Landgren [EMAIL PROTECTED] wrote: I know I had my eyes opened by Devel::Cover. I thought I had pretty good coverage in Regexp::Assemble. In fact I had about 60%. I lifted it up to 100% statement coverage (some branching and conditional paths are never taken, but they are normal, for example C$foo or return 0). This toook about 15 hours of work over a week or so. I caught a couple of minor bugs and one rather big one in the process. Well, I dont think it would be possible for me to get 100% coverage in a module like Data::Dump::Streamer. Im pretty happy that I managed to get about 80% actually. The problem for me is that i have a fair amount of code that is specifically for one version of perl, or another, or to handle never happen cases (like Scalar::Util::reftype() returning a unknown type). Likewise all kinds of things that are supported in 5.8 are totally meaningless in 5.6.x (locked hashes anyone?). OTOH, i was happy that DC illustrated some dead codepaths that probably could be removed. But im resigned to never getting 100%. And, it doesnt help that something about DC breaks the defined operator when dealing with overloaded objects. (yeah, he did say the code was alpha quality :-) cheers, yves -- perl -Mre=debug -e /just|another|perl|hacker/
Re: New kwalitee test, has_changes
demerphq wrote: On 9/21/05, David Landgren [EMAIL PROTECTED] wrote: I know I had my eyes opened by Devel::Cover. I thought I had pretty good coverage in Regexp::Assemble. In fact I had about 60%. I lifted it up to 100% statement coverage (some branching and conditional paths are never taken, but they are normal, for example C$foo or return 0). This toook about 15 hours of work over a week or so. I caught a couple of minor bugs and one rather big one in the process. Well, I dont think it would be possible for me to get 100% coverage in a module like Data::Dump::Streamer. Im pretty happy that I managed to get about 80% actually. The problem for me is that i have a fair amount of code that is specifically for one version of perl, or another, or to handle never happen cases (like Scalar::Util::reftype() returning a unknown type). Likewise all kinds of things that are supported in 5.8 are totally meaningless in 5.6.x (locked hashes anyone?). You miss my point. Whether the code be cross-platform or cross-version, you need to aggregate the coverage results from all the environments your code is designed to run on. can't happen is another kettle of fish, of course. I wouldn't advocate stripping out defensive coding in order to improve coverage. But then again, in large system failures it's the never-visted code, executed in failure modes, that are regularly singled out as a source of failure amplification. Just read computer.risks for a while. David OTOH, i was happy that DC illustrated some dead codepaths that probably could be removed. But im resigned to never getting 100%. And, it doesnt help that something about DC breaks the defined operator when dealing with overloaded objects. (yeah, he did say the code was alpha quality :-) cheers, yves -- It's overkill of course, but you can never have too much overkill.
Re: New kwalitee test, has_changes
David Landgren wrote: demerphq wrote: You miss my point. Whether the code be cross-platform or cross-version, you need to aggregate the coverage results from all the environments your code is designed to run on. How is this done?
Re: New kwalitee test, has_changes
Hi!
On Fri, Sep 16, 2005 at 09:23:10AM +1000, Adam Kennedy wrote:
Yeah, something like that. Changes, for a suitably flexible value of
Changes
I implemented this in CPANTS. It will be in the next release (tomorrow),
results should be available on Sunday morning.
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$-gprint$_.$/}
Re: New kwalitee test, has_changes
Thomas Klausner writes: On Thu, Sep 15, 2005 at 11:52:00AM +1000, Adam Kennedy wrote: Rather than do any additional exploding, I'd like to propose the additional kwalitee test has_changes. I've noticed that a percentage (5-10%) of dists don't have a changes file, so it can be hard to know whether it's worth upgrading, or more importantly which version to add dependencies for. I like it So do I. But I think the name is misleading. When I read the subject and first two sentences of Adam's proposal I thought has changes would be a test for actually having changes, rather than for having a file which lists changes. Calling the test has_changelog would be much better, since that makes it obvious was is being tested. For a start, I will only check for a file called /changes/i. After the run, we'll see how many percent use it and try to find a regex that matches more dists. As others have pointed out, ChangeLog is also popular and given special treatment by Cpan Search; please allow CHANGES or ChangeLog. Smylers -- May God bless us with enough foolishness to believe that we can make a difference in this world, so that we can do what others claim cannot be done.
Re: New kwalitee test, has_changes
Adam Kennedy wrote: Rather than do any additional exploding, I'd like to propose the additional kwalitee test has_changes. I've noticed that a percentage (5-10%) of dists don't have a changes file, so it can be hard to know whether it's worth upgrading, or more importantly which version to add dependencies for. Adam K Would this look for Change OR ChangeLog? Both seem to be popular on CPAN. -=Chris smime.p7s Description: S/MIME Cryptographic Signature
Re: New kwalitee test, has_changes
On 9/15/05, Christopher H. Laco [EMAIL PROTECTED] wrote: Would this look for Change OR ChangeLog? Both seem to be popular on CPAN. And some people put their changelog in the README. Gabor
Re: New kwalitee test, has_changes
Ricardo SIGNES wrote: * Christopher H. Laco [EMAIL PROTECTED] [2005-09-15T08:23:57] Would this look for Change OR ChangeLog? Both seem to be popular on CPAN. ...and some modules have a HISTORY or CHANGES section of POD, and DBI has DBI::Changes. Though, as with pod and pod coverage tests, a Changes file with See DBI::Changes or See Readme would satisfy the kwalitee test. It a hack to satisfy the metric, but has the side benefit of actually providing a consistent place for someone to track down the actual location of the change log. Regards, David Golden
Re: New kwalitee test, has_changes
On 9/15/05, David Landgren [EMAIL PROTECTED] wrote: As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80%, and another for 100%, and how absolutely impossible it would be to set out to establish these points for all the modules on CPAN. But it would be Good. I have to wonder about how you handle modules that have code that is Perl version dependent or OS dependent. Many non-trivial modules end up having to work around one such incompatibility or another, and therefore can't get full coverage on a single perl/OS combination. So to be fair about it you'd have to aggregate the coverage for all the combinations of OS/Perl version to find out how the cover goes. -- perl -Mre=debug -e /just|another|perl|hacker/
Re: New kwalitee test, has_changes
On Thu, 2005-09-15 at 16:00 +0200, David Landgren wrote: As I was downloading the newest version of Devel::Cover this morning, I pondered on the concept of 1 Kwalitee point for coverage = 80%, and another for 100%, and how absolutely impossible it would be to set out to establish these points for all the modules on CPAN. But it would be Good. 80% is probably fine, but I see 100% test coverage as indicating that the module is either simple enough not to trip bugs or really difficult problems in Devel::Cover or that the author has mangled the code sufficiently to get 100% coverage. The former might be a sign of kwalitee, but the latter really just isn't. -- c
Re: New kwalitee test, has_changes
Ricardo SIGNES wrote: * Christopher H. Laco [EMAIL PROTECTED] [2005-09-15T08:23:57] Would this look for Change OR ChangeLog? Both seem to be popular on CPAN. ...and some modules have a HISTORY or CHANGES section of POD, and DBI has DBI::Changes. Good point. Modules created with ExtUtils::ModuleMaker, by default, include a CHANGES file in the distribution. But it's easy to set an option to include the changes in the HISTORY section of the POD. Someone who does that should not be viewed as creating a distribution of lesser quality than someone who follows the default. TMTOWTDI. jimk
