The Lancaster Consensus from this year's QA hackathon has updated the desired semantics: http://www.dagolden.com/index.php/2098/the-annotated-lancaster-consensus/
Specifically, AUTOMATED_TESTING is now intended to distinguish only between things running for "smoke testing" versus things running as part of module installation (or other, manually testing). It no longer means "without recourse to a human operator" as it did after the Oslo Consensus. That aspect is now covered by the new "NONINTERACTIVE_TESTING" environment variable. In your case -- automated testing with manual review of results -- both should be set, because when the tests run, it is both smoking and non-interactive. :-) Your manual review is awesome -- I wish more smoke testers did that -- but it doesn't change the environment we want to present to distributions, so they can decide to opt-in or out of some or all tests depending on circumstances surrounding the test. David On Thu, Jun 13, 2013 at 5:19 PM, Slaven Rezic <sla...@rezic.de> wrote: > Karen Etheridge <p...@froods.org> writes: > >> Hi Slaven, >> >> recently I received a number of failing test reports from you, for >> Plack-App-BeanstalkConsole-0.004: >> >> http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290 >> http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613 >> http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8 >> http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2 >> http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3 >> >> However, there should be *no testing at all* done for this distribution, >> (save any reports that are sent manually via cpanm-reporter), because the >> first line of Makefile.PL is: >> >> exit 0 if $ENV{AUTOMATED_TESTING}; >> >> This is because this distribution requires the presence of a php >> executable, and also the Makefile.PL downloads a large file from github -- >> so to respect github's servers, I'd rather this happen only when the module >> is actually going to be used. >> >> Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable: >> >> http://wiki.cpantesters.org/wiki/CPANAuthorNotes >> >> "Can I tell if my module is being tested by an automated client (a.k.a. >> 'smoker')?" >> >> Automated smoke testers should set $ENV{AUTOMATED_TESTING} to a true >> value. This allows authors to skip certain tests (or include certain tests) >> when the results are not being monitored by a human being. >> >> One could even go so far as to halt building and testing a distribution >> under automated testing by exiting with zero at the top of the Makefile.PL >> or Build.PL file: >> >> exit 0 if $ENV{AUTOMATED_TESTING}; >> >> >> Please could you set AUTOMATED_TESTING in your smokers? thank you! >> > > Actually, my smoker does not fail under this sentence: "... are not > being monitored by a human being". I look into every FAIL report before > sending. Really. > > But I can setup a special rule in 01.DISABLED.yml for your distribution. > > Regards, > Slaven > > -- > Slaven Rezic - slaven <at> rezic <dot> de > BBBike - route planner for cyclists in Berlin > WWW version: http://www.bbbike.de > Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net -- David Golden <x...@xdg.me> Take back your inbox! → http://www.bunchmail.com/ Twitter/IRC: @xdg
