Re: [BackupPC-users] What's using all this disk space?
On Mon, Oct 04, 2010 at 03:32:23PM -0400, Timothy J Massey wrote:
Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010
03:28:23 PM:
On Mon, Oct 04, 2010 at 03:25:03PM -0400, Timothy J Massey
wrote:
Robin Lee Powell rlpow...@digitalkingdom.org wrote on
10/04/2010 03:15:29 PM:
How do I find out which backups are using a lot of disk?
We'd like to see if there's a problem with our retention
policy, especially on database servers, but I've no insight
at all into where all this disk is *going*.
Anyone got a script for this?
I don't have a script for this, but if you look at the host
page for each server, examine the New Files section. This
will tell you which backups are consuming a lot of space (i.e.
aren't pooling well).
We have 200+ servers getting backed up on here. :)
Well, then, you'll want to parse the pc/hostname/backups file.
The 9th (New Files Count) and 10th (New Files Size) field (AFAICT)
are what you're looking for.
Sorry, no script.
I've got one. Attached. Specialized for our environment, not
productionalized or anything, but it works.
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which this parrot
is dead is ti poi spitaki cu morsi, but this sentence is false
is na nei. My personal page: http://www.digitalkingdom.org/rlp/
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
chdir '/backups/pc';
opendir(DIR, '.') || die can't opendir . $!;
my @dirs = grep { ! /^\./ } readdir(DIR);
closedir DIR;
my %values;
foreach my $dir (@dirs) {
open( BACKUPS, $dir/backups ) || do {
print Could not open file /backups/pc/$dir/backups \n;
next;
};
my $total_total=0;
my $new_total=0;
my $num_backups=0;
while( BACKUPS ) {
my @fields = split( /\t/ );
if( $fields[5] =~ /^\d+$/ $fields[9] =~ /^\d+$/ ) {
# THe total size for this backup
$total_total += $fields[5];
# Minus what was already there
$total_total -= $fields[7];
# THe new/additional size for this backup
$new_total += $fields[9];
$num_backups++;
}
}
close( BACKUPS );
$values{$dir} = {
total = $total_total,
new = $new_total,
num = $num_backups,
}
}
print q{
Total New
SizeAverage New Size Per Backup
Backup NameTotal Size
Number Of Backups
};
foreach my $key (sort { $values{$b}-{total} = $values{$a}-{total} } keys
%values) {
# printf( Backup $key has %20.2d GiB of backups on disk total.\n, (
($values{$key}-{total}) / 1024 / 1024 / 1024 ) );
my $total_gib=( ($values{$key}-{total}) / 1024 / 1024 / 1024 );
my $new_gib=( ($values{$key}-{new}) / 1024 / 1024 / 1024 );
my $num_backups=$values{$key}-{num};
my $avg_new_mib=( ( ($values{$key}-{new}) / $num_backups ) / 1024 / 1024 /
1024 );
format STDOUT =
@ @.## GiB @.##
GiB @## @. GiB
$key, $total_gib, $new_gib, $num_backups, $avg_new_mib
.
write;
}
--
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] What's using all this disk space?
On Mon, Oct 04, 2010 at 03:25:03PM -0400, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:15:29 PM: How do I find out which backups are using a lot of disk? We'd like to see if there's a problem with our retention policy, especially on database servers, but I've no insight at all into where all this disk is *going*. Anyone got a script for this? I don't have a script for this, but if you look at the host page for each server, examine the New Files section. This will tell you which backups are consuming a lot of space (i.e. aren't pooling well). We have 200+ servers getting backed up on here. :) Good to know where to look, though. Database servers have the same problem that mail servers have: large files that change each and every single day, and therefore consume their full amount of space for each backup you keep. *nod* -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] What's using all this disk space?
Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:15:29 PM: How do I find out which backups are using a lot of disk? We'd like to see if there's a problem with our retention policy, especially on database servers, but I've no insight at all into where all this disk is *going*. Anyone got a script for this? I don't have a script for this, but if you look at the host page for each server, examine the New Files section. This will tell you which backups are consuming a lot of space (i.e. aren't pooling well). For example, on your mail servers, we will have New Files of tens of gigabytes for each backup. Let's say that the entire server uses 100GB of space, of which 60GB is mail data. If we keep 12 copies around, we would consume about 760GB (100GB for the server as a whole plus 60 * 11 additional copies). A different server, such as a file server, might be 500GB big, but the New Files section might be, on average, under 1GB. In that case, those same 12 copies would only take 512GB. Even though the server is 5 times larger, the *deltas* are significantly smaller, and the total space used is smaller, too. Database servers have the same problem that mail servers have: large files that change each and every single day, and therefore consume their full amount of space for each backup you keep. Tim Massey -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] What's using all this disk space?
On Mon, Oct 04, 2010 at 03:32:23PM -0400, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:28:23 PM: On Mon, Oct 04, 2010 at 03:25:03PM -0400, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:15:29 PM: How do I find out which backups are using a lot of disk? We'd like to see if there's a problem with our retention policy, especially on database servers, but I've no insight at all into where all this disk is *going*. Anyone got a script for this? I don't have a script for this, but if you look at the host page for each server, examine the New Files section. This will tell you which backups are consuming a lot of space (i.e. aren't pooling well). We have 200+ servers getting backed up on here. :) Well, then, you'll want to parse the pc/hostname/backups file. The 9th (New Files Count) and 10th (New Files Size) field (AFAICT) are what you're looking for. *Ooooh*. I thought I had to write Perl to talk to the server. That's *easy*. Thank you! -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] What's using all this disk space?
Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:45:11 PM: On Mon, Oct 04, 2010 at 03:32:23PM -0400, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:28:23 PM: On Mon, Oct 04, 2010 at 03:25:03PM -0400, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 10/04/2010 03:15:29 PM: How do I find out which backups are using a lot of disk? We'd like to see if there's a problem with our retention policy, especially on database servers, but I've no insight at all into where all this disk is *going*. Anyone got a script for this? I don't have a script for this, but if you look at the host page for each server, examine the New Files section. This will tell you which backups are consuming a lot of space (i.e. aren't pooling well). We have 200+ servers getting backed up on here. :) Well, then, you'll want to parse the pc/hostname/backups file. The 9th (New Files Count) and 10th (New Files Size) field (AFAICT) are what you're looking for. *Ooooh*. I thought I had to write Perl to talk to the server. That's *easy*. You don't have to write any Perl to talk to the server: Craig has done an outstanding job of packaging the functions of BackupPC into powerful command line functions. Virtually anything you can do from the GUI (and more!) can be done from the command line, using *any* language you want. In addition, BackupPC stores everything you see in the GUI in nicely formatted text files that can be processed however you would like. The most important one being the backups file in each host's directory. I'm glad that pointing that out helped! Tim Massey -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
