Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Leon Timmermans]

On Wed, Mar 29, 2017 at 8:37 PM, Dan Collins  wrote:
> Huh, I just found another interesting thing. CPAN::Reporter makes no effort
> to report on whether the `install` phase succeeded, but some particularly
> creative dists manage to fail in the `install` phase under
> PERL_USE_UNSAFE_INC=0. Look at this example from
> GUIDO/libintl-perl-1.26.tar.gz - does anyone know how common this sort of
> thing is? There aren't many using this exact name
> (http://grep.cpan.me/?q=package+MyInstall).
>
> All tests successful.
> Files=164, Tests=3750,  5 wallclock secs ( 0.87 usr  0.18 sys +  8.23 cusr
> 0.30 csys =  9.58 CPU)
> Result: PASS
> (/usr/bin/make test exited with 0)
> CPAN::Reporter: Test result is 'pass', 'make test' no errors.
> CPAN::Reporter: preparing a CPAN Testers report for libintl-perl-1.26
>
> CPAN::Reporter: this appears to be a duplicate report for the test phase:
> PASS libintl-perl-1.26 x86_64-linux-thread-multi 3.16.0-4-amd64
>
> Test report will not be sent.
>
>   GUIDO/libintl-perl-1.26.tar.gz
>   /usr/bin/make test -- OK
> Running make install
> make[1]: Entering directory
> '/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs'
> "/home/cpan2/install/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' --
> gettext_xs.bs ../blib/arch/auto/Locale/gettext_xs/gettext_xs.bs 644
> make[1]: Leaving directory
> '/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs'
> Manifying 30 pod documents
> Manifying 29 pod documents
> Manifying 31 pod documents
> Manifying 29 pod documents
> Manifying 30 pod documents
> Can't locate MyInstall.pm in @INC (you may need to install the MyInstall
> module) (@INC contains:
> /home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/arch
> /home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/lib
> /home/cpan2/install/lib/perl5/site_perl/5.26.0/x86_64-linux-thread-multi
> /home/cpan2/install/lib/perl5/site_perl/5.26.0
> /home/cpan2/install/lib/perl5/5.26.0/x86_64-linux-thread-multi
> /home/cpan2/install/lib/perl5/5.26.0).
> BEGIN failed--compilation aborted.
> Makefile:1389: recipe for target 'pure_site_install' failed
> make: *** [pure_site_install] Error 2
>   GUIDO/libintl-perl-1.26.tar.gz
>   /usr/bin/make install  -- NOT OK
> Stopping: 'install' failed for 'Locale::TextDomain'.

I've seen the blib/arch issue before, but never this solution to it.

Leon

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Andreas Koenig]

> On Wed, 29 Mar 2017 14:37:30 -0400, Dan Collins  
> said:

  > Huh, I just found another interesting thing. CPAN::Reporter makes no
  > effort to report on whether the `install` phase succeeded, but some
  > particularly creative dists manage to fail in the `install` phase
  > under PERL_USE_UNSAFE_INC=0.

Related: https://github.com/gfx/p5-Mouse/issues/74 and the latest three
postings on https://github.com/toddr/perl/issues/14

-- 
andreas

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Dan Collins]

Huh, I just found another interesting thing. CPAN::Reporter makes no effort
to report on whether the `install` phase succeeded, but some particularly
creative dists manage to fail in the `install` phase under
PERL_USE_UNSAFE_INC=0. Look at this example
from GUIDO/libintl-perl-1.26.tar.gz - does anyone know how common this sort
of thing is? There aren't many using this exact name (
http://grep.cpan.me/?q=package+MyInstall).

All tests successful.
Files=164, Tests=3750,  5 wallclock secs ( 0.87 usr  0.18 sys +  8.23 cusr
 0.30 csys =  9.58 CPU)
Result: PASS
(/usr/bin/make test exited with 0)
CPAN::Reporter: Test result is 'pass', 'make test' no errors.
CPAN::Reporter: preparing a CPAN Testers report for libintl-perl-1.26

CPAN::Reporter: this appears to be a duplicate report for the test phase:
PASS libintl-perl-1.26 x86_64-linux-thread-multi 3.16.0-4-amd64

Test report will not be sent.

  GUIDO/libintl-perl-1.26.tar.gz
  /usr/bin/make test -- OK
Running make install
make[1]: Entering directory
'/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs'
"/home/cpan2/install/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' --
gettext_xs.bs ../blib/arch/auto/Locale/gettext_xs/gettext_xs.bs 644
make[1]: Leaving directory
'/home/cpan2/.cpan/build/libintl-perl-1.26-55/gettext_xs'
Manifying 30 pod documents
Manifying 29 pod documents
Manifying 31 pod documents
Manifying 29 pod documents
Manifying 30 pod documents
Can't locate MyInstall.pm in @INC (you may need to install the MyInstall
module) (@INC contains:
/home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/arch
/home/cpan2/.cpan/build/libintl-perl-1.26-55/blib/lib
/home/cpan2/install/lib/perl5/site_perl/5.26.0/x86_64-linux-thread-multi
/home/cpan2/install/lib/perl5/site_perl/5.26.0
/home/cpan2/install/lib/perl5/5.26.0/x86_64-linux-thread-multi
/home/cpan2/install/lib/perl5/5.26.0).
BEGIN failed--compilation aborted.
Makefile:1389: recipe for target 'pure_site_install' failed
make: *** [pure_site_install] Error 2
  GUIDO/libintl-perl-1.26.tar.gz
  /usr/bin/make install  -- NOT OK
Stopping: 'install' failed for 'Locale::TextDomain'.

On Wed, Mar 29, 2017 at 2:25 PM, Dan Collins  wrote:

> Porters, I heard you wanted some red and yellow...
>
> I have tooling set up that allows me to test/install most of CPAN, saving
> the CPAN::Reporter results to file, and compare those stored files between
> two different versions of perl. I installed two instances of perl - cpan1
> and cpan2 - which are completely identical. For one of the environments, I
> set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined.
>
> I'm not done yet, but I have over 5,000 dists that have been tested on
> both configurations. Of those dists, I've collected the reports that PASSED
> with PERL_USE_UNSAFE_INC=1 and FAILED or were UNKNOWN with
> PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at:
> http://dcollins.cc/p5p.html
>
> Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means
> they failed in tests, the latter means they failed in Build.PL or
> Makefile.PL. Of course, some particularly special dists had multiple
> problems. There are a few cases where a dist FAILed in the control: 5 where
> it PASSed in the experiment, which are probably signs of an intermittent
> test failure and which should be ignored, and 6 where it got UNKNOWN in the
> experiment, which are probably signs of a no-dot-in-inc error *in addition
> to* a test failure.
>
> I haven't uploaded these test reports yet, because I want to include a
> message in the "Tester Comments" explaining the failures that module
> authors might be able to use to fix their stuff. I'll be watching the other
> thread that Kent started to see when that resource is available
>
> Happy to answer any questions or take any suggestions...
>
> --
> Dan
>
>
> On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots 
> wrote:
>
>> I've gone and built a naive version of this for my own uses.  It doesn't
>> yet hook into cpanm-reporter but i don't think it'd be hard to do.  It's
>> intending to take a --module or --cpanfile to start from and works its way
>> through the dependency tree (fetches it all with cpanm --showdeps) and then
>> tests them with perlbrew exec --with ... so that the script can run in a
>> stable perl install while testing current blead/git easily.  I'll toss it
>> up on github if there's any big call for it (otherwise it'll just live
>> publicly on my server).
>>
>> https://gitea.simcop2387.info/simcop2387/treedeps
>>
>> On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric 
>> wrote:
>>
>>> On 27 March 2017 at 16:10, Dan Collins  wrote:
>>> > I've had a chance to make sure that this test environment isn't
>>> horribly
>>> > broken...
>>>
>>>
>>> Yeah, there's enough broken you sort of need a double-testing system.
>>>
>>> Start with =0 , and then upon hitting a failure, retry with =1 ...
>>> while paying care
>>> 

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Dan Collins]

Porters, I heard you wanted some red and yellow...

I have tooling set up that allows me to test/install most of CPAN, saving
the CPAN::Reporter results to file, and compare those stored files between
two different versions of perl. I installed two instances of perl - cpan1
and cpan2 - which are completely identical. For one of the environments, I
set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined.

I'm not done yet, but I have over 5,000 dists that have been tested on both
configurations. Of those dists, I've collected the reports that PASSED with
PERL_USE_UNSAFE_INC=1 and FAILED or were UNKNOWN with
PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at:
http://dcollins.cc/p5p.html

Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means
they failed in tests, the latter means they failed in Build.PL or
Makefile.PL. Of course, some particularly special dists had multiple
problems. There are a few cases where a dist FAILed in the control: 5 where
it PASSed in the experiment, which are probably signs of an intermittent
test failure and which should be ignored, and 6 where it got UNKNOWN in the
experiment, which are probably signs of a no-dot-in-inc error *in addition
to* a test failure.

I haven't uploaded these test reports yet, because I want to include a
message in the "Tester Comments" explaining the failures that module
authors might be able to use to fix their stuff. I'll be watching the other
thread that Kent started to see when that resource is available

Happy to answer any questions or take any suggestions...

--
Dan


On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots 
wrote:

> I've gone and built a naive version of this for my own uses.  It doesn't
> yet hook into cpanm-reporter but i don't think it'd be hard to do.  It's
> intending to take a --module or --cpanfile to start from and works its way
> through the dependency tree (fetches it all with cpanm --showdeps) and then
> tests them with perlbrew exec --with ... so that the script can run in a
> stable perl install while testing current blead/git easily.  I'll toss it
> up on github if there's any big call for it (otherwise it'll just live
> publicly on my server).
>
> https://gitea.simcop2387.info/simcop2387/treedeps
>
> On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric 
> wrote:
>
>> On 27 March 2017 at 16:10, Dan Collins  wrote:
>> > I've had a chance to make sure that this test environment isn't horribly
>> > broken...
>>
>>
>> Yeah, there's enough broken you sort of need a double-testing system.
>>
>> Start with =0 , and then upon hitting a failure, retry with =1 ...
>> while paying care
>> not to let the =1 bleed into children in the recursive step.
>>
>> eg:
>>
>> 1. Install X with UNSAFE_INC=0
>> 2. Configure fails?
>> 1. Report N/A Fail
>> 2. Generate META.JSON with UNSAFE_INC=1
>> 3. Installdeps with UNSAFE_INC=0
>> 4. Run tests with UNSAFE_INC=0
>> 5. Tests Fail?
>> 1. Report FAIL
>> 2. Re-run  tests with UNSAFE_INC=1
>> 3. Report Test result
>>
>> I don't know how to make my tools do this automatically so I've been
>> doing it manually and filing bugs, to fill the gaps in automation.
>>
>>
>>
>> --
>> Kent
>>
>> KENTNL - https://metacpan.org/author/KENTNL
>>
>
>

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Ryan Voots]

I've gone and built a naive version of this for my own uses.  It doesn't
yet hook into cpanm-reporter but i don't think it'd be hard to do.  It's
intending to take a --module or --cpanfile to start from and works its way
through the dependency tree (fetches it all with cpanm --showdeps) and then
tests them with perlbrew exec --with ... so that the script can run in a
stable perl install while testing current blead/git easily.  I'll toss it
up on github if there's any big call for it (otherwise it'll just live
publicly on my server).

https://gitea.simcop2387.info/simcop2387/treedeps

On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric  wrote:

> On 27 March 2017 at 16:10, Dan Collins  wrote:
> > I've had a chance to make sure that this test environment isn't horribly
> > broken...
>
>
> Yeah, there's enough broken you sort of need a double-testing system.
>
> Start with =0 , and then upon hitting a failure, retry with =1 ...
> while paying care
> not to let the =1 bleed into children in the recursive step.
>
> eg:
>
> 1. Install X with UNSAFE_INC=0
> 2. Configure fails?
> 1. Report N/A Fail
> 2. Generate META.JSON with UNSAFE_INC=1
> 3. Installdeps with UNSAFE_INC=0
> 4. Run tests with UNSAFE_INC=0
> 5. Tests Fail?
> 1. Report FAIL
> 2. Re-run  tests with UNSAFE_INC=1
> 3. Report Test result
>
> I don't know how to make my tools do this automatically so I've been
> doing it manually and filing bugs, to fill the gaps in automation.
>
>
>
> --
> Kent
>
> KENTNL - https://metacpan.org/author/KENTNL
>

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Alceu Rodrigues de Freitas Junior via cpan-testers-discuss]

Hello Kent,

I didn't know this environment variable, but I understand the concern.

On the other hand, it doesn't seem to be a easy thing to fix:

http://www.nntp.perl.org/group/perl.perl5.porters/2016/11/msg241020.html
https://www.effectiveperlprogramming.com/2017/01/v5-26-removes-dot-from-inc/

Instead of trying to setup the environment, isn't possible to workaround 
and systematically try to identify the problem instead of causing it? 
I'm thinking about a standard unit test that we could incorporate into 
CPAN::Reporter.


Regards,
Alceu

Em 27/03/2017 00:39, Kent Fredric escreveu:

On 27 March 2017 at 16:10, Dan Collins  wrote:

I've had a chance to make sure that this test environment isn't horribly
broken...



Yeah, there's enough broken you sort of need a double-testing system.

Start with =0 , and then upon hitting a failure, retry with =1 ...
while paying care
not to let the =1 bleed into children in the recursive step.

eg:

1. Install X with UNSAFE_INC=0
2. Configure fails?
1. Report N/A Fail
2. Generate META.JSON with UNSAFE_INC=1
3. Installdeps with UNSAFE_INC=0
4. Run tests with UNSAFE_INC=0
5. Tests Fail?
1. Report FAIL
2. Re-run  tests with UNSAFE_INC=1
3. Report Test result

I don't know how to make my tools do this automatically so I've been
doing it manually and filing bugs, to fill the gaps in automation.

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Kent Fredric]

On 27 March 2017 at 16:10, Dan Collins  wrote:
> I've had a chance to make sure that this test environment isn't horribly
> broken...


Yeah, there's enough broken you sort of need a double-testing system.

Start with =0 , and then upon hitting a failure, retry with =1 ...
while paying care
not to let the =1 bleed into children in the recursive step.

eg:

1. Install X with UNSAFE_INC=0
2. Configure fails?
1. Report N/A Fail
2. Generate META.JSON with UNSAFE_INC=1
3. Installdeps with UNSAFE_INC=0
4. Run tests with UNSAFE_INC=0
5. Tests Fail?
1. Report FAIL
2. Re-run  tests with UNSAFE_INC=1
3. Report Test result

I don't know how to make my tools do this automatically so I've been
doing it manually and filing bugs, to fill the gaps in automation.



-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: Shortage of Red / Yellow on CPAN Testers Matrix

[Dan Collins]

I started the process of running a cpan smoke test for this and noticed
that even CPAN::Reporter and YAML::Syck fail under PERL_USE_UNSAFE_INC=0.
I'll start uploading reports and sharing results with the list once I've
had a chance to make sure that this test environment isn't horribly
broken...


On Sun, Mar 26, 2017 at 10:05 PM, Kent Fredric 
wrote:

> TL;DR:
> - Testers, please set PERL_USE_UNSAFE_INC=0 to make the bugs actually show
> up.
> - P5P: I don't know what to say, but testers aren't yet to really even
> start showing the real . in @INC Bugs, please proceed with caution.
>
> 
>
> I'm getting a little afraid that not enough testers are running in
> configurations that expose '.' in @INC errors.
>
> I had expected to have the Testers Matrixes filled with reds and
> yellows already, but it seems whenever I stumble into a "this package
> is broken with '.' in @INC", I look in the matrix, and all I see is
> green.
>
> This means all 3 classes of scope problems are hidden.
>
> - Makefile.PL failures are missing
> - t/  failures are missing
>
> and importantly:
>
> - tests exposing runtime bugs are missing.
>
> Because setting '.' by default in the test phases not only makes the
> tests pass superficially,
> but hides the runtime bugs.
>
> For example, Data::FormValidator has a runtime bug:
>
> https://rt.cpan.org/Ticket/Display.html?id=120671
>
> And yet:
>
> http://fast-matrix.cpantesters.org/?dist=Data-FormValidator+4.85
>
> I'm the only person of the many who tested it to spot the failure that
> affects 5.26
>
> Essentially, it seems the Majority of testers right now are repressing
> a serious failure path in their tester stacks.
>
> Here's something further upriver:
>
> http://fast-matrix.cpantesters.org/?dist=MRO-Compat+0.12
>
> Makefile.PL broken "because MI".
>
> This is a real problem that affects downstream installations and
> vendors.( Even though vendors will do their own testing/stuff, seeing
> the bugs upstream before they percolate through the ecosystem is very
> much useful )
>
> But so far I'm the only one who had a stack setup to error. ( And it
> wasn't even hard: I'm just using cpanm )
>
> --
> Kent
>
> KENTNL - https://metacpan.org/author/KENTNL
>