Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

Thanks,

I committed the attached patch to fix the issue.

Jean-Louis


On 16/11/17 11:08 AM, Nathan Stratton Treadway wrote:
> On Thu, Nov 16, 2017 at 09:43:15 -0500, Jean-Louis Martineau wrote:
> > On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> > > DUMP SUMMARY:
> > > DUMPER STATS TAPER STATS
> > > HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> > > --- --- --
> > > --
> > It is a bug if they are not listed, can you send me the
> > log..0 file
> >
>
> I have just sent this log file in a separate (off-list) message.
>
> Let me know if you need any other info.
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail
diff --git a/installcheck/Amanda_Report.pl b/installcheck/Amanda_Report.pl
index bea9f05..ceef57d 100755
--- a/installcheck/Amanda_Report.pl
+++ b/installcheck/Amanda_Report.pl
@@ -28,6 +28,7 @@ use Data::Dumper;
 use lib '@amperldir@';
 
 use Installcheck;
+use Installcheck::Run;
 use Amanda::Config qw( :init :getconf );
 use Amanda::Report;
 use Amanda::Debug;
@@ -41,6 +42,9 @@ my %LogfileData;
 my $logCount = 0;
 my $log_filename = "$Installcheck::TMP/Amanda_Report_test.log";
 
+my $testconf = Installcheck::Run::setup();;
+$testconf->write();
+
 # copy/pasted from installcheck/Amanda_Logfile.pl .  Maybe move this
 # to a module?
 sub write_logfile
diff --git a/perl/Amanda/Report.pm b/perl/Amanda/Report.pm
index 643ea07..5f4ed40 100644
--- a/perl/Amanda/Report.pm
+++ b/perl/Amanda/Report.pm
@@ -932,7 +932,6 @@ sub _handle_dumper_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data = $self->{data};
-my $disklist = $data->{disklist};
 my $programs = $data->{programs};
 my $dumper_p = $programs->{dumper} ||= {};
 
@@ -953,7 +952,7 @@ sub _handle_dumper_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $orig_kb =~ s{\]$}{};
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $self->{'run_timestamp'});
 my $dumper = $try->{dumper} ||= {};
 	$dumper->{level} = $level;
@@ -992,7 +991,7 @@ sub _handle_dumper_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $orig_kb =~ s{\]$}{};
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $timestamp );
 my $dumper = $try->{dumper} ||= {};
 
@@ -1009,7 +1008,7 @@ sub _handle_dumper_line
 my @info = Amanda::Util::split_quoted_strings($str);
 my ( $hostname, $disk, $timestamp, $level ) = @info[ 0 .. 3 ];
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $timestamp );
 my $dumper = $try->{dumper} ||= {};
 	$dumper->{date}  = $timestamp;
@@ -1040,7 +1039,6 @@ sub _handle_chunker_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data  = $self->{data};
-my $disklist  = $data->{disklist};
 my $programs  = $data->{programs};
 my $chunker_p = $programs->{chunker} ||= {};
 
@@ -1061,7 +1059,7 @@ sub _handle_chunker_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $kps =~ s{\]$}{};
 
-my $dle = $disklist->{$hostname}->{$disk};
+my $dle = $self->_get_disklist($hostname, $disk);
 my $try = $self->_get_try( $dle, "chunker", $timestamp );
 my $chunker = $try->{chunker} ||= {};
 
@@ -1094,7 +1092,6 @@ sub _handle_taper_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data = $self->{data};
-my $disklist = $data->{disklist};
 my $programs = $data->{programs};
 my $taper_p  = $programs->{taper} ||= {};
 
@@ -1161,7 +1158,7 @@ sub _handle_taper_line
 $kps =~ s{\]$}{};
 $orig_kb =~ s{\]$}{} if defined($orig_kb);
 
-my $dle   = $disklist->{$hostname}{$disk};
+my $dle   = $self->_get_disklist($hostname, $disk);
 my $try   = $self->_get_try($dle, "taper", $timestamp);
 my $taper = $try->{taper} ||= {};
 my $parts = $taper->{parts} ||= [];
@@ -1232,7 +1229,7 @@ sub _handle_taper_line
 $kps 

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Fri, Nov 17, 2017 at 09:10:05 -0500, Jean-Louis Martineau wrote:
> I committed the attached patch to fix the issue.

Yep, with that patch the report now shows the VAULT lines as expected
(when run against the log file from that original "amdump" run).

Thanks.

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 10:15:28 -0500, Jean-Louis Martineau wrote:
> Nathan,
> 
> Try this patch.
> 
> run: amdump --no-dump --no-flush CONFIG
> Will do all scheduled vault to the vault-storage

[...]

> diff --git a/server-src/driver.c b/server-src/driver.c
> index 351494a..fd01296 100644
> --- a/server-src/driver.c
> +++ b/server-src/driver.c
> @@ -107,6 +107,10 @@ static int taper_started = 0;
>  static int nb_storage;
>  static cmddatas_t *cmddatas = NULL;
>  
> +static int no_dump = FALSE;
> +//static int no_flush = FALSE;
> +static int no_vault = FALSE;
> +
>  static int wait_children(int count);
>  static void wait_for_children(void);
>  static void allocate_bandwidth(netif_t *ip, unsigned long kps);
> @@ -305,6 +309,30 @@ main(
>  }
>  
>  if (argc > 2) {
> + if (g_str_equal(argv[2], "--no-dump")) {
> + no_dump = TRUE;
> + argv++;
> + argc--;
> + }
> +}

I noticed when reviewing the log file from my "amdump --no-dump" run
that there were still "INFO dumper" lines in there, from when the
dumpers start up at the beginning of the run and when they terminate at
the end (with no activity in between).

I wondered if the dumpers should be getting started at all when no_dump
is true, and took a look at driver.c to see how the dumpers were getting
kicked off -- and discovered that driver.c already implements a "nodump"
option, and that indeed if that option is enabled, the dumper processes
are not spawned.

(I see that this "nodump" option is used by Amflush.pm.)

Anyway, I guess the question is whether it makes sense to merge the new
--no-dump and original nodump options for driver.c, at least as far as
having one internal boolean variable to handle both -- or, if not,
if there are any places where the code currently checks for !nodump
where !no_dump should be treated equivalently?

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 10:15:28 -0500, Jean-Louis Martineau wrote:
> Nathan,
> 
> Try this patch.
> 
> run: amdump --no-dump --no-flush CONFIG
> Will do all scheduled vault to the vault-storage

I recompiled with this patch applied and repeated the test.  Sure
enough, the above command did do the vaulting for all the dumps saved in
the first phase, and the report it generated did not have the 
  planner: FATAL no DLE to backup
or
  planner: Argument 'BADHOST' matches neither a host nor a disk.
  driver: WARNING: got empty schedule from planner
lines... 

(Also, the DUMP SUMMARY section did included the VAULT operation for
each dump, as expected after the other Report-3.5.diff patch.)


The one thing that wasn't quite correct this time around was that the
FAILURE DUMP SUMMARY section of the report *did* have a RESULTS MISSING
line for each DLE.

After comparing the log files for the run from the original email and
this one, I believe these messages are triggered because using --no-dump
means the log file has a "DISK planner" line for each DLE in the
disklist (lines which don't exist when BADHOST is given).  

(But I'm not sure if the correct solution is for  no_dump = TRUE to also
also suppress those "DISK planner" lines from being generated in the
first place, or if Report.pm should treat a VAULT line in the DUMP
SUMMARY as a valid "result" for that DLE...or something else.)


Nathan



Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239