RE: Devel::Cover better practice to identify uncovered files

[Jason Pyeron]

> -Original Message-
> From: James E Keenan
> Sent: Tuesday, April 18, 2017 22:28
> 
> On 04/18/2017 07:43 PM, Jason Pyeron wrote:
> > Currently we are using this script to find files (line 48) 
> under the scripts directory and add them to the coverage report.
> >
> > After running the programs under test, the script is run:
> >
> > $ ./tests/script/cover-missing.pl
> > load
> > ingest
> > 17 known covered file(s) found
> > preprocess
> > 132 uncovered file(s) found and hashed
> > process
> > run: 1492558405..10744
> > saving
> >
> > Here it found over 100 files without executions, so our 
> tests are not very comprehensive...
> >
> > First question, is the code below the "best" way to do this?
> >
> > Second question, is this something that could be provided 
> by the Module via API or command line? Something like: cover 
> -know fileName . Maybe even support if recursively find files 
> if fileName is a directory.
> >
> 
> Can I ask you to back up a minute and tell us how you "normally" test 
> your code -- i.e., test it in the absence of coverage analysis?

Sure. It is not tested. 

Your intentions are right, but not aligned to the practical issues. I am 
working on a massive update to a long running open source project. The code 
base does not have ANY tests when I started. I am changing much of the 
rendering engine, but preserving legacy functionality. To assure everyone, we 
are creating tests in parallel, not before.

Code coverage will tell us how comprehensive our test are. Right now they suck. 
Nothing helps more than having a finidh line. This does that.


> 
> Testing of *programs* -- what I presume you have under your scripts/ 
> directory -- is somewhat different.  Testing a script from 
> inside a Perl 
> test script is just a special case of running any program 
> from inside a 

What one has to understand is that the code was written to load perl files 
dynamically, not all branches load all files. Here in lies the rub.


> 
> Now, clearly, your mileage may vary.  That's why I'm interested in 
> knowing your concerns about doing coverage analysis on the programs 
> under scripts/.

I sort of mentioned this above. I am adding XML output to logwatch. Logwatch 
uses an execution model that loads perl files dynamically. Further we do not 
have anywhere sufficient test coverage at this point, but that can no longer be 
a blocker. The coverage reports should indicate zero coverage on files which 
have not been loaded / run. Having 78% of 2% of the files is not 78% it is 
closer to 1%.

The more important issue is the future, when new log processors are added the 
code coverage in Jenkins will catch it by seeing a new file that is never 
called, reporting the code coverage falling below the threshold.

-Jason

Devel::Cover better practice to identify uncovered files

[Jason Pyeron]

Currently we are using this script to find files (line 48) under the scripts 
directory and add them to the coverage report. 

After running the programs under test, the script is run:

$ ./tests/script/cover-missing.pl
load
ingest
17 known covered file(s) found
preprocess
132 uncovered file(s) found and hashed
process
run: 1492558405..10744
saving

Here it found over 100 files without executions, so our tests are not very 
comprehensive...

First question, is the code below the "best" way to do this?

Second question, is this something that could be provided by the Module via API 
or command line? Something like: cover -know fileName . Maybe even support if 
recursively find files if fileName is a directory.

$ cat -n tests/script/cover-missing.pl [source at 
https://sourceforge.net/u/jpyeron/logwatch/ci/master/tree/tests/script/cover-missing.pl]
 1  #!/usr/bin/perl -w
 2
 3  use Data::Dumper;
 4  use File::Find;
 5  use Cwd;
 6
 7  print "load\n";
 8
 9  use Devel::Cover::DB;
10
11  my $dbpath="cover_db";
12  my $db = Devel::Cover::DB->new(db => $dbpath);
13  my $timeStart=time;
14  my $runKey="$timeStart..$$";
15  my %known;
16
17  print "ingest\n";
18
19  find({ wanted => \_coverfile, no_chdir => 1 }, "$dbpath/runs/");
20  find({ wanted => \_coverfile, no_chdir => 1 }, 
"$dbpath/digests");
21  find({ wanted => \_coverfile, no_chdir => 1 }, 
"$dbpath/structure/");
22
23  sub process_coverfile
24  {
25  if (-f $_)
26  {
27  my $x=$db->read($_);
28  foreach my $run ($x->runs)
29  {
30  my $h=$run->{digests};
31  foreach my $file (keys %$h)
32  {
33  if ( ! exists $known{$file} )
34  {
35  $known{$file}=$run->{digests}{$file};
36  }
37  }
38  }
39  }
40  }
41
42  print scalar keys %known, " known covered file(s) found\n";
43
44  print "preprocess\n";
45
46  my %toadd;
47
48  find({ wanted => \_file, no_chdir => 1 }, "scripts");
49
50  sub process_file
51  {
52  if (-f $_)
53  {
54  if ( ! exists $known{$_} )
55  {
56  $toadd{$_}=Devel::Cover::DB::Structure->digest($_);
57  }
58  }
59  }
60
61  print scalar keys %toadd, " uncovered file(s) found and hashed\n";
62
63
64  print "process\n";
65
66  if (scalar keys %toadd == 0)
67  {
68  print "no files to process\n";
69  exit;
70  }
71
72  print "run: $runKey\n";
73
74  $db->{runs}{$runKey}{"OS"}=$^O;
75  
$db->{runs}{$runKey}{"collected"}=["branch","condition","pod","statement","subroutine","time"];
76  $db->{runs}{$runKey}{"dir"}=Cwd::abs_path();
77  $db->{runs}{$runKey}{"vec"}={};
78  $db->{runs}{$runKey}{"start"}=$timeStart;
79  $db->{runs}{$runKey}{"run"}=$0;
80  $_=$^V;
81  s/v//;
82  $db->{runs}{$runKey}{"perl"}=$_;
83
84  my $s=$db->{structure}=Devel::Cover::DB::Structure->new;
85
86  foreach my $file (keys %toadd)
87  {
88  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"branch"}=undef;
89  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"condition"}=undef;
90  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"pod"}=undef;
91  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"subroutine"}=undef;
92  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"time"}=undef;
93  
$db->{structure}->{f}{$file}{start}{-1}{"__COVER__"}[0]{"statement"}=0;
94  $db->{structure}->{f}{$file}{file}=$file;
95  $db->{structure}->{f}{$file}{digest}=$toadd{$file};
96  $db->{structure}->{f}{$file}{statement}=[1];
97  $db->{runs}{$runKey}{"count"}{$file}{'statement'}=[0];
98  $db->{runs}{$runKey}{"digests"}{$file}=$toadd{$file};
99  }
   100
   101  $db->{runs}{$runKey}{"finish"}=time;
   102
   103  print "saving\n";
   104
   105  $db->write("$dbpath/runs/$runKey");


-Jason

RE: Cevel::Cover missing lines?

[Jason Pyeron]

> -Original Message-
> From: Paul Johnson 
> Sent: Tuesday, April 18, 2017 15:29
> 
> On Tue, Apr 18, 2017 at 12:32:57PM -0400, Jason Pyeron wrote:
> > I am "require"ing a file but Devel::Cover is not logging 
> the statements, only the sub, use, and eval portions.
> > 
> > I tried a do(filename.pl) to no help.
> > 
> > I tried adding -subs_only off (and on) to no effect.
> > 
> > Any suggestions?
> 
> The problem here is the require or do.  After running the top-level
> statements, perl throws them away, which means that by the time
> Devel::Cover comes to match up the data it has recorded against the 
> ops in the program it can't find the ops for those top-level statements.

That is sad.

> 
> I had originally thought it would be quite easy to persuade 
> perl not to
> throw away such statements, but it has proven beyond my abilities so
> far.
> 
> The toolchain summit is coming up shortly and I may see if I can't
> impose on one of the clever folk who will be attending for a little
> assistance.

That would be awesome. How can I help?

> 
> So, my only practical suggestion, I'm afraid, is to bundle all the
> top-level statements into a subroutine (which, as an old C 
> programmer I
> tend to call "main") and then call that subroutine as the 
> only top-level
> statement.

Changing the the code to support this is not an option.

-Jason

Cevel::Cover missing lines?

[Jason Pyeron]

I am "require"ing a file but Devel::Cover is not logging the statements, only 
the sub, use, and eval portions.

I tried a do(filename.pl) to no help.

I tried adding -subs_only off (and on) to no effect.

Any suggestions?

(source can be found at 
https://sourceforge.net/u/jpyeron/logwatch/ci/master/tree/)

-Jason
Title: File Coverage: scripts/logwatch.pl





File Coverage

File:scripts/logwatch.pl
Coverage:48.1%



linestmtbrancondsubpodtimecode
1#!/usr/bin/perl -w
225252525004000use strict;
3##
4##
5# Most current version can always be found at:
6# ftp://ftp.logwatch.org/pub/linux (tarball)
7# ftp://ftp.logwatch.org/pub/redhat/RPMS (RPMs)
89
10# Specify version and build-date:
11my $Version = '7.4.3';
12my $VDate = '04/27/16';
1314###
15# Logwatch was written and is maintained by:
16#Kirk Bauer 
17#
18# Unless otherwise specified, Logwatch and all bundled filter scripts
19# are Copyright (c) Kirk Bauer and covered under the included MIT/X
20# Consortium license.
21#
22# Please send all comments, suggestions, bug reports,
23#etc, to logwa...@logwatch.org.
24#
25
2627
28# ENV SETTINGS:
29# About the locale:  some functions use locale information.  In particular,
30# Logwatch makes use of strftime, which makes use of LC_TIME variable.  Other
31# functions may also use locale information.
32#
33# Because the parsing must be in the same locale as the logged information,
34# and this appears to be "C", "POSIX", or "en_US", we set LC_ALL for
35# this and other scripts invoked by this script.  We use "C" because it
36# is always (?) available, whereas POSIX or en_US may not.  They all use
37# the same time formats and rely on the ASCII character set.
38#
39# Variables REAL_LANG and REAL_LC_ALL keep the original values for use by
40# scripts that need native language.
41$ENV{'REAL_LANG'}=$ENV{'LANG'} if $ENV{'LANG'};
42$ENV{'REAL_LC_ALL'}=$ENV{'LC_ALL'} if $ENV{'LC_ALL'};
4344# Setting ENV for scripts invoked by this script.
45$ENV{'LC_ALL'} = "C";
46# Using setlocale to set locale for this script.
4725252525440032120110use POSIX qw(locale_h);
48setlocale(LC_ALL, "C");
4950my $BaseDir = "/usr/share/logwatch";
51my $ConfigDir = "/etc/logwatch";
52my $PerlVersion = "$^X";
5354#
5556#
57# SET LIBS, GLOBALS, and DEFAULTS
5825252525870022550141000use Getopt::Long;
592525252520use POSIX qw(uname);
602525252557003284021339017use File::Temp qw/ tempdir /;
6162my (%Config, @ServiceList, @LogFileList, %ServiceData, %LogFileData);
63my (@AllShared, @AllLogFiles, @FileList);
64# These need to not be global variables one day
65my (@ReadConfigNames, @ReadConfigValues);
6667my @argvClone = @ARGV;
68my $gop = Getopt::Long::Parser->new;
69$gop->configure("pass_through");
70$gop->getoptionsfromarray(\@argvClone, 
71  "confdir=s" => \$ConfigDir,
72  "basedir=s" => \$BaseDir,
73  "debug=s"   => \$Config{'debug'},
74 );
75762525252542002160021000eval "use lib \"$BaseDir/lib\";";
772525252514400604000eval "use Logwatch \':dates\'";
7879# Default config here...
80$Config{'detail'} = 0;
81# if MAILTO is set in the environment, grab it, as it may be used by cron
82# or anacron
83if ($ENV{'MAILTO'}) {
84   $Config{'mailto'} = $ENV{'MAILTO'};
85} else {
86   $Config{'mailto'} = "root";
87}
88$Config{'mailfrom'} = "Logwatch";
89$Config{'subject'} = "";
Snipped to make file smaller
1527sub output {
1528179817981000   my ($index, $text, $type) = @_;
1529   #Types are start stop header line ref
153015311798502000   if ( $type eq "ref_extra" ) {
153200  $out_reference .= "$text\n";
1533   }
153415351798502000   if ( $type eq "ref" ) {
153600  $out_reference .= "   
1537   }
1538153917981001000   if ( $type eq "start" ) {
1540711000  $reports[$index] = "$text";
1541   #SERVICE table headers if ( $index eq 'E' ) { #never happens change out_body from hash back to array
154271502000  if ( $Config{'format'} eq "html" ) {
154300 $out_body{$index} .=
1544 "
1545 
1546   
1547   $reports[$index]
1548   \n";
1549  }
1550   }
1551155217981001000   if ( $type eq "stop" ) {
155371500  if ( $Config{'format'} eq "html" ) {
155400 $out_body{$index} .= "  \n";
155500 $out_body{$index} .= "  \n";
1556  }
1557   }
155815591798503000   if ( $type eq "header" ) {
1560   if ( $Config{'format'} eq "text" ) {

RE: Devel::Cover force adding unseen files - feedback requested.

[Jason Pyeron]

> -Original Message-
> From: Tina Muller 
> Sent: Monday, December 26, 2016 07:04
> 
> Hi Jason,
> 
> On Sun, 25 Dec 2016, Jason Pyeron wrote:
> 
> > Coverage works great as part on continuous integration, 
> > until a new file is added and the unit tests are blissfully 
> > ignorant of the new file's existence.
> 
> Maybe this is not what you're looking for, but it works well
> for me loading any module in t/00.load.t:
> 
>  use Test::More;
> 
>  # unorthodox but works fine
>  use Module::Pluggable search_path => ['Module::Namespace'];
>  my @plugins = main->plugins;
>  for my $module (@plugins) {
>  use_ok( $module );
>  }
> 
>  done_testing;

Cute way to search, thanks.



> 
> Of course, this only works for .pm files.

Here in lies the rub, the files (e.g. 
https://sourceforge.net/p/logwatch/git/ci/master/tree/scripts/) are just plain 
old perl files.

-Jason