Huh, I just found another interesting thing. CPAN::Reporter makes no effort to report on whether the `install` phase succeeded, but some particularly creative dists manage to fail in the `install` phase under PERL_USE_UNSAFE_INC=0. Look at this example from GUIDO/libintl-perl-1.26.tar.gz - does anyone know how common this sort of thing is? There aren't many using this exact name ( http://grep.cpan.me/?q=package+MyInstall).
All tests successful. Files=164, Tests=3750, 5 wallclock secs ( 0.87 usr 0.18 sys + 8.23 cusr 0.30 csys = 9.58 CPU) Result: PASS (/usr/bin/make test exited with 0) CPAN::Reporter: Test result is 'pass', 'make test' no errors. CPAN::Reporter: preparing a CPAN Testers report for libintl-perl-1.26 CPAN::Reporter: this appears to be a duplicate report for the test phase: PASS libintl-perl-1.26 x86_64-linux-thread-multi 3.16.0-4-amd64 Test report will not be sent. GUIDO/libintl-perl-1.26.tar.gz /usr/bin/make test -- OK Running make install make[1]: Entering directory '/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs' "/home/cpan2/install/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- gettext_xs.bs ../blib/arch/auto/Locale/gettext_xs/gettext_xs.bs 644 make[1]: Leaving directory '/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs' Manifying 30 pod documents Manifying 29 pod documents Manifying 31 pod documents Manifying 29 pod documents Manifying 30 pod documents Can't locate MyInstall.pm in @INC (you may need to install the MyInstall module) (@INC contains: /home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/arch /home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/lib /home/cpan2/install/lib/perl5/site_perl/5.26.0/x86_64-linux-thread-multi /home/cpan2/install/lib/perl5/site_perl/5.26.0 /home/cpan2/install/lib/perl5/5.26.0/x86_64-linux-thread-multi /home/cpan2/install/lib/perl5/5.26.0). BEGIN failed--compilation aborted. Makefile:1389: recipe for target 'pure_site_install' failed make: *** [pure_site_install] Error 2 GUIDO/libintl-perl-1.26.tar.gz /usr/bin/make install -- NOT OK Stopping: 'install' failed for 'Locale::TextDomain'. On Wed, Mar 29, 2017 at 2:25 PM, Dan Collins <dcolli...@gmail.com> wrote: > Porters, I heard you wanted some red and yellow... > > I have tooling set up that allows me to test/install most of CPAN, saving > the CPAN::Reporter results to file, and compare those stored files between > two different versions of perl. I installed two instances of perl - cpan1 > and cpan2 - which are completely identical. For one of the environments, I > set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined. > > I'm not done yet, but I have over 5,000 dists that have been tested on > both configurations. Of those dists, I've collected the reports that PASSED > with PERL_USE_UNSAFE_INC=1 and FAILED or were UNKNOWN with > PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at: > http://dcollins.cc/p5p.html > > Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means > they failed in tests, the latter means they failed in Build.PL or > Makefile.PL. Of course, some particularly special dists had multiple > problems. There are a few cases where a dist FAILed in the control: 5 where > it PASSed in the experiment, which are probably signs of an intermittent > test failure and which should be ignored, and 6 where it got UNKNOWN in the > experiment, which are probably signs of a no-dot-in-inc error *in addition > to* a test failure. > > I haven't uploaded these test reports yet, because I want to include a > message in the "Tester Comments" explaining the failures that module > authors might be able to use to fix their stuff. I'll be watching the other > thread that Kent started to see when that resource is available > > Happy to answer any questions or take any suggestions... > > -- > Dan > > > On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots <simcop2...@simcop2387.info> > wrote: > >> I've gone and built a naive version of this for my own uses. It doesn't >> yet hook into cpanm-reporter but i don't think it'd be hard to do. It's >> intending to take a --module or --cpanfile to start from and works its way >> through the dependency tree (fetches it all with cpanm --showdeps) and then >> tests them with perlbrew exec --with ... so that the script can run in a >> stable perl install while testing current blead/git easily. I'll toss it >> up on github if there's any big call for it (otherwise it'll just live >> publicly on my server). >> >> https://gitea.simcop2387.info/simcop2387/treedeps >> >> On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric <kentfred...@gmail.com> >> wrote: >> >>> On 27 March 2017 at 16:10, Dan Collins <dcolli...@gmail.com> wrote: >>> > I've had a chance to make sure that this test environment isn't >>> horribly >>> > broken... >>> >>> >>> Yeah, there's enough broken you sort of need a double-testing system. >>> >>> Start with =0 , and then upon hitting a failure, retry with =1 ... >>> while paying care >>> not to let the =1 bleed into children in the recursive step. >>> >>> eg: >>> >>> 1. Install X with UNSAFE_INC=0 >>> 2. Configure fails? >>> 1. Report N/A Fail >>> 2. Generate META.JSON with UNSAFE_INC=1 >>> 3. Installdeps with UNSAFE_INC=0 >>> 4. Run tests with UNSAFE_INC=0 >>> 5. Tests Fail? >>> 1. Report FAIL >>> 2. Re-run tests with UNSAFE_INC=1 >>> 3. Report Test result >>> >>> I don't know how to make my tools do this automatically so I've been >>> doing it manually and filing bugs, to fill the gaps in automation. >>> >>> >>> >>> -- >>> Kent >>> >>> KENTNL - https://metacpan.org/author/KENTNL >>> >> >> >
