There are a number of issues in the parrot distribution that deviate from assumptions about a how a "normal" CPAN distribution behaves and those cause CPAN::Reporter's heuristics to mis-grade it.
It gets a PASS because it looks like the distribution has a recursive Makefile.PL (ext/Parrot-Embed/Makefile.PL), so we can't easily disambiguate the test harness output and instead rely on the exit code only -- and parrot uses a custom harness program (t/harness) that doesn't return a non-zero exit code for failing tests. (I.e. "make test" has exit code 0 even with failing tests.) So -- things that CPAN::Reporter could do: * tighten the check for recursive Makefile.PL to only include the first level of sub-directories below the main one * in the case of a recursive Makefile.PL, parse the output anyway and do something smart. There was a reason that I avoided the second approach when dealing with recursive Makefile.PL's, but I don't recall exactly what it is, now. CPAN::Reporter parses the output in reverse order of lines looking for a Test::Harness result line and maybe I didn't want to risk scanning all of the test output in case some distribution's test output happened to resemble Test::Harness output. That's a little less risky with the newer Test::Harness that has a single, well defined format for output. On the other hand, I don't really think we should be testing parrot with CPAN Testers anyway, so this isn't a high priority for me to fix. I'm willing to do the first option I gave, above, because it's simple and should break anything else. (Unless someone convinces me it's a bad idea). -- David On Sat, Oct 25, 2008 at 4:23 AM, Andreas J. Koenig <[EMAIL PROTECTED]> wrote: > (CC'd to testers-discuss, maybe somebody has some insight?) > > Hi David, > > it looks like a bug in CPAN::Reporter to me (please make your mail reader > wide): > > % /home/src/perl/repoperls/installed-perls/*/p*/[EMAIL PROTECTED]/bin/perl -I > lib bin/ctgetreports --q meta:writer --q meta:from --q meta:perl --q > qr:"(Failed test.*)" --ctformat=yaml --vdistro=parrot-0.8.0 parrot |& grep > 'Failed test ' > PASS 2485750 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:[EMAIL PROTECTED] qr:(Failed > test.*)[Failed test 'pasm_output_isnt: failure'] > PASS 2473167 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:[EMAIL PROTECTED] qr:(Failed > test.*)[Failed test 'soup to nuts'] > PASS 2472586 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:perl[5.8.7] qr:(Failed test.*)[Failed > test 'pasm_output_isnt: failure'] > PASS 2472469 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:perl[5.8.8] qr:(Failed test.*)[Failed > test 'pasm_output_isnt: failure'] > PASS 2472396 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:[EMAIL PROTECTED] qr:(Failed > test.*)[Failed test 'pasm_output_isnt: failure'] > PASS 2472319 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:perl[5.10.0] qr:(Failed > test.*)[Failed test 'pasm_output_isnt: failure'] > PASS 2472246 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:[EMAIL PROTECTED] qr:(Failed > test.*)[Failed test 'soup to nuts'] > PASS 2472202 meta:writer[CPAN-Reporter-1.1702] > meta:from[andreas.koenig.gmwojprw] meta:perl[5.10.0] qr:(Failed > test.*)[Failed test 'pasm_output_isnt: failure'] > PASS 2471971 meta:writer[CPAN-Reporter-1.1702] meta:from[imacat] > meta:perl[5.10.0] qr:(Failed test.*)[Failed test 'pasm_output_isnt: failure'] > > > What you see here is that all my reports and one from imacat reported a PASS > but the output of the test contained "Failed test '....'". > > When reading these reports from top to bottom I find no clue why this > happens. Do you? > > -- > andreas >
