Porters, I heard you wanted some red and yellow... I have tooling set up that allows me to test/install most of CPAN, saving the CPAN::Reporter results to file, and compare those stored files between two different versions of perl. I installed two instances of perl - cpan1 and cpan2 - which are completely identical. For one of the environments, I set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined.
I'm not done yet, but I have over 5,000 dists that have been tested on both configurations. Of those dists, I've collected the reports that PASSED with PERL_USE_UNSAFE_INC=1 and FAILED or were UNKNOWN with PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at: http://dcollins.cc/p5p.html Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means they failed in tests, the latter means they failed in Build.PL or Makefile.PL. Of course, some particularly special dists had multiple problems. There are a few cases where a dist FAILed in the control: 5 where it PASSed in the experiment, which are probably signs of an intermittent test failure and which should be ignored, and 6 where it got UNKNOWN in the experiment, which are probably signs of a no-dot-in-inc error *in addition to* a test failure. I haven't uploaded these test reports yet, because I want to include a message in the "Tester Comments" explaining the failures that module authors might be able to use to fix their stuff. I'll be watching the other thread that Kent started to see when that resource is available Happy to answer any questions or take any suggestions... -- Dan On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots <simcop2...@simcop2387.info> wrote: > I've gone and built a naive version of this for my own uses. It doesn't > yet hook into cpanm-reporter but i don't think it'd be hard to do. It's > intending to take a --module or --cpanfile to start from and works its way > through the dependency tree (fetches it all with cpanm --showdeps) and then > tests them with perlbrew exec --with ... so that the script can run in a > stable perl install while testing current blead/git easily. I'll toss it > up on github if there's any big call for it (otherwise it'll just live > publicly on my server). > > https://gitea.simcop2387.info/simcop2387/treedeps > > On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric <kentfred...@gmail.com> > wrote: > >> On 27 March 2017 at 16:10, Dan Collins <dcolli...@gmail.com> wrote: >> > I've had a chance to make sure that this test environment isn't horribly >> > broken... >> >> >> Yeah, there's enough broken you sort of need a double-testing system. >> >> Start with =0 , and then upon hitting a failure, retry with =1 ... >> while paying care >> not to let the =1 bleed into children in the recursive step. >> >> eg: >> >> 1. Install X with UNSAFE_INC=0 >> 2. Configure fails? >> 1. Report N/A Fail >> 2. Generate META.JSON with UNSAFE_INC=1 >> 3. Installdeps with UNSAFE_INC=0 >> 4. Run tests with UNSAFE_INC=0 >> 5. Tests Fail? >> 1. Report FAIL >> 2. Re-run tests with UNSAFE_INC=1 >> 3. Report Test result >> >> I don't know how to make my tools do this automatically so I've been >> doing it manually and filing bugs, to fill the gaps in automation. >> >> >> >> -- >> Kent >> >> KENTNL - https://metacpan.org/author/KENTNL >> > >
