Re: [Bacula-users] file listing?
I wrote a script to do this for me, I call it bacula-du, since it
accepts many of the same options as du(1), and the output is the same.
Usage: bacula-du [OPTIONS] -j JOBID
Summarize disk usage of directories included in the backup JOBID
Options are:
-a, --all write counts for all files, not just directories
-b, --bytes use size in octets rather than number of blocks
-B, --block-size=SIZE report SIZE-byte blocks (default 1Ki)
-mlike --block-size=1Mi
-S, --separate-dirs do not include size of subdirectories
-t, --threshold=SIZE skip output for files or directories with usage
below SIZE
-L, --largest=NUM only print NUM largest directories/files
SIZE may be (or may be an integer optionally followed by) one of following:
k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi.
I hope others can find it useful.
--
Kjetil T. Homme
Redpill Linpro AS - Changing the game
#! /usr/bin/perl -w
# bacula-du 1.0
# Written by Kjetil Torgrim Homme kjetil.ho...@redpill-linpro.com
# Released under GPLv3 or the same terms as Bacula itself
sub usage {
print _END_;
Usage: $0 [OPTIONS] -j JOBID
Summarize disk usage of directories included in the backup JOBID
Options are:
-a, --all write counts for all files, not just directories
-b, --bytes use size in octets rather than number of blocks
-B, --block-size=SIZE report SIZE-byte blocks (default 1Ki)
-mlike --block-size=1Mi
-S, --separate-dirs do not include size of subdirectories
-t, --threshold=SIZE skip output for files or directories with usage
below SIZE
-L, --largest=NUM only print NUM largest directories/files
SIZE may be (or may be an integer optionally followed by) one of following:
k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi.
_END_
exit(64);
}
use strict;
use DBD::mysql;
use DBI;
use MIME::Base64;
use Getopt::Long qw(:config bundling no_ignore_case);
use Data::Dumper;
my $dbhost = localhost;
my $db = bacula;
my $dsn = DBI:Pg:dbname=$db;host=$dbhost;
my $dbuser = postgres;
my $dbpass = ;
# Suggestion for MySQL:
# my $dsn = DBI:mysql:database=mysql;mysql_read_default_group=clientp;
# my $dbuser = mysql;
# my $dbpass = undef;
###
my $i = 0;
my %base64 = map { $_ = $i++ } split(,
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/);
sub decode_bacula_base64 {
my $acc = 0;
for (split(, $_[0])) {
$acc = 6;
$acc += $base64{$_};
}
return $acc;
}
sub extract_size_from_lstat {
return decode_bacula_base64((split( , shift))[7]);
}
sub extract_blocks_from_lstat {
return 512 * decode_bacula_base64((split( , shift))[9]);
}
sub convert_units {
my $num = shift;
my %units = (k = 1000**1, Ki = 1024**1, ki = 1024**1,
M = 1000**2, Mi = 1024**2,
G = 1000**3, Gi = 1024**3,
T = 1000**4, Ti = 1024**4,
P = 1000**5, Pi = 1024**5);
if ($num =~ /^(\d*)([kKMGTP]i?)B?$/) {
$num = ($1 ? $1 : 1) * $units{$2};
} elsif ($num !~ /^\d+$/) {
die Can't parse: $num\n;
}
return $num;
}
### main program resumes
my $threshold = 1; # omit 0 octet sized files/directories by default
my $blocksize = 1024;
my ($jobid, $all, $bytes, $separate_dirs, $largest);
GetOptions(jobid|j=i = \$jobid,
threshold|t=s = \$threshold,
separate-dirs|S = \$separate_dirs,
all|a = \$all,
bytes|b = \$bytes,
block-size|B=s = \$blocksize,
largest|L=i = \$largest,
m = sub { $blocksize = 1Mi },
) || usage();
usage() unless $jobid;
$threshold = convert_units($threshold);
$blocksize = convert_units($blocksize);
my @padding = (, A==, ==, =);
sub extract_size_from_lstat_foo {
my ($b64) = (split( , shift))[7];
my $acc = 0;
for (split(, decode_base64($b64 . $padding[length($b64) % 4]))) {
$acc = 8;
$acc += ord($_);
}
return $acc;
}
my $extract_size = $bytes
? \extract_size_from_lstat
: \extract_blocks_from_lstat;
my $dbh;
unless ($dbh = DBI-connect($dsn, $dbuser, $dbpass, {AutoCommit = 0})) {
print STDERR Could not connect to database $db on host $dbhost\n;
exit 2;
}
my $sth = $dbh-prepare(
SELECT p.Path, fn.Name, LStat
FROM Path p
JOIN File f ON f.PathId = p.PathId
JOIN Filename fn ON f.FilenameId = fn.FilenameId
WHERE f.JobId = $jobid);
$sth-execute();
my %du;
my $rowcount = 0;
while (my ($path, $fname, $lstat) = $sth-fetchrow_array) {
my $size = $extract_size-($lstat);
# print STDERR Got '$path' size $size\n;
$du{$path$fname} += $size if $all;
$du{$path} += $size;
next if $separate_dirs;
while ($path ne '/') {
$path =~ s,[^/]+/$,,;
$du{$path} += $size;
}
if ((++$rowcount % 1000) == 0) {
print STDERR got $rowcount
Re: [Bacula-users] file listing?
On 03/15/11 12:41, Kjetil Torgrim Homme wrote: I wrote a script to do this for me, I call it bacula-du, since it accepts many of the same options as du(1), and the output is the same. Usage: bacula-du [OPTIONS] -j JOBID Summarize disk usage of directories included in the backup JOBID Options are: -a, --all write counts for all files, not just directories -b, --bytes use size in octets rather than number of blocks -B, --block-size=SIZE report SIZE-byte blocks (default 1Ki) -mlike --block-size=1Mi -S, --separate-dirs do not include size of subdirectories -t, --threshold=SIZE skip output for files or directories with usage below SIZE -L, --largest=NUM only print NUM largest directories/files SIZE may be (or may be an integer optionally followed by) one of following: k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi. I hope others can find it useful. Kjetil, Thanks, this looks like a useful tool. I'm forwarding your post to the bacula-devel list to make sure folks there see it too. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, SQL wrangler, Free Stater It's not the years, it's the mileage. -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On 3/15/2011 12:41 PM, Kjetil Torgrim Homme wrote: I wrote a script to do this for me, I call it bacula-du, since it accepts many of the same options as du(1), and the output is the same. Usage: bacula-du [OPTIONS] -j JOBID Summarize disk usage of directories included in the backup JOBID Options are: -a, --all write counts for all files, not just directories -b, --bytes use size in octets rather than number of blocks -B, --block-size=SIZE report SIZE-byte blocks (default 1Ki) -mlike --block-size=1Mi -S, --separate-dirs do not include size of subdirectories -t, --threshold=SIZE skip output for files or directories with usage below SIZE -L, --largest=NUM only print NUM largest directories/files SIZE may be (or may be an integer optionally followed by) one of following: k (1000), Ki (1024), M (1000*1000), Mi (1024*1024), G, Gi, T, Ti, P, Pi. I hope others can find it useful. Can you give us an example of the output please? -- Dan Langille - http://langille.org/ -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On 3/12/2011 9:33 PM, ganiuszka wrote:
W dniu 13 marca 2011 01:01 użytkownik Dan Langilled...@langille.org
napisał:
On 3/12/2011 9:29 AM, ganiuszka wrote:
W dniu 11 marca 2011 21:45 użytkownik ganiuszkaganius...@gmail.com
napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this eight:
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] =89
[inodes] =1366434
[mode] =33200
[links] =1
[uid] =80
[gid] =91
[rdev] =5481898
[size] =315
[blksize] =16384
[blocks] =4
[atime] =1299770037
[mtime] =1299770037
[ctime] =1299770066
[LinkFl] =0
[flags] =0
[data] =2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langilled...@langille.org:
On 3/9/2011 5:36 PM, ganiuszka wrote:
2011/3/9 Mike Egglestonmikee...@mac.com:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried
'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the size
of the file?
Mike
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 :=
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) *
(64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please make
sure that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
$blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' = $dev, 'inode' = $inode, 'mode'
=
$mode, 'links' = $links, 'uid' = $uid, 'gid' = $gid, 'rdev' =
$rdev, 'size' = $size, 'blksize' = $blksize, 'blocks' = $blocks,
'atime' = $atime, 'mtime' = $mtime, 'ctime' = $ctime, 'linkfi' =
$linkfi, 'flags' = $flags, 'data' = $data);
$ret = array();
foreach($encoded_values as $key = $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i strlen($val); $i++) {
$result= 6;
$result += strpos($base64, substr($val, $i , 1));
}
$ret[$key] = ($is_minus === true) ? -$result : $result;
}
return $ret;
}
and here is usage for this:
$lstat = 'gB DL+b IGg B A y A D5dZR BAA fN4 BNeR+z BNeR+7 BNeR+7 A A C';
$decoded_lstat = decode_bacula_lstat($lstat);
print_r($decoded_lstat);
Any suggestions and modifications are welcome.
Regards
gani
I tried this function. I got an
Re: [Bacula-users] file listing?
Hi,
Thank you for feedback again.
Yes, the BCMath module is not in default configuration if you will
compile PHP. In case compilation from sources need to add
--enable-bcmath to configuration parameters.
On some GNU/Linux distributions the BCMath is as default in PHP
(Debian, Slackware) and on some distributions the BCMath need to
install as a module from package apart from the PHP package (for
example Fedora).
Regards.
gani
W dniu 13 marca 2011 21:07 użytkownik Dan Langille d...@langille.org napisał:
On 3/12/2011 9:33 PM, ganiuszka wrote:
W dniu 13 marca 2011 01:01 użytkownik Dan Langilled...@langille.org
napisał:
On 3/12/2011 9:29 AM, ganiuszka wrote:
W dniu 11 marca 2011 21:45 użytkownik ganiuszkaganius...@gmail.com
napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this eight:
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] = 89
[inodes] = 1366434
[mode] = 33200
[links] = 1
[uid] = 80
[gid] = 91
[rdev] = 5481898
[size] = 315
[blksize] = 16384
[blocks] = 4
[atime] = 1299770037
[mtime] = 1299770037
[ctime] = 1299770066
[LinkFl] = 0
[flags] = 0
[data] = 2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langilled...@langille.org:
On 3/9/2011 5:36 PM, ganiuszka wrote:
2011/3/9 Mike Egglestonmikee...@mac.com:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried
'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the
size
of the file?
Mike
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 :=
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) *
(64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please
make
sure that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
$blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' = $dev, 'inode' = $inode,
'mode' =
$mode, 'links' = $links, 'uid' = $uid, 'gid' = $gid, 'rdev' =
$rdev, 'size' = $size, 'blksize' = $blksize, 'blocks' = $blocks,
'atime' = $atime, 'mtime' = $mtime, 'ctime' = $ctime, 'linkfi' =
$linkfi, 'flags' = $flags, 'data' = $data);
$ret = array();
foreach($encoded_values as $key = $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i strlen($val); $i++) {
Re: [Bacula-users] file listing?
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please make sure
that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size, $blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' = $dev, 'inode' = $inode, 'mode' =
$mode, 'links' = $links, 'uid' = $uid, 'gid' = $gid, 'rdev' =
$rdev, 'size' = $size, 'blksize' = $blksize, 'blocks' = $blocks,
'atime' = $atime, 'mtime' = $mtime, 'ctime' = $ctime, 'linkfi' =
$linkfi, 'flags' = $flags, 'data' = $data);
$ret = array();
foreach($encoded_values as $key = $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i strlen($val); $i++) {
$result = 6;
$result += strpos($base64, substr($val, $i , 1));
}
$ret[$key] = ($is_minus === true) ? -$result : $result;
}
return $ret;
}
and here is usage for this:
$lstat = 'gB DL+b IGg B A y A D5dZR BAA fN4 BNeR+z BNeR+7 BNeR+7 A A C';
$decoded_lstat = decode_bacula_lstat($lstat);
print_r($decoded_lstat);
Any suggestions and modifications are welcome.
Regards
gani
W dniu 11 marca 2011 21:45 użytkownik ganiuszka ganius...@gmail.com napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this eight:
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] = 89
[inodes] = 1366434
[mode] = 33200
[links] = 1
[uid] = 80
[gid] = 91
[rdev] = 5481898
[size] = 315
[blksize] = 16384
[blocks] = 4
[atime] = 1299770037
[mtime] = 1299770037
[ctime] = 1299770066
[LinkFl] = 0
[flags] = 0
[data] = 2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langille d...@langille.org:
On 3/9/2011 5:36 PM, ganiuszka wrote:
2011/3/9 Mike Egglestonmikee...@mac.com:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried 'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the size
of the file?
Mike
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
--
Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie
za przyjaciół swoich. Jezus Chrystus
--
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the
Re: [Bacula-users] file listing?
On 3/12/2011 9:29 AM, ganiuszka wrote:
W dniu 11 marca 2011 21:45 użytkownik ganiuszkaganius...@gmail.com napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this eight:
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] = 89
[inodes] = 1366434
[mode] = 33200
[links] = 1
[uid] = 80
[gid] = 91
[rdev] = 5481898
[size] = 315
[blksize] = 16384
[blocks] = 4
[atime] = 1299770037
[mtime] = 1299770037
[ctime] = 1299770066
[LinkFl] = 0
[flags] = 0
[data] = 2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langilled...@langille.org:
On 3/9/2011 5:36 PM, ganiuszka wrote:
2011/3/9 Mike Egglestonmikee...@mac.com:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried 'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the size
of the file?
Mike
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please make sure
that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size, $blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' = $dev, 'inode' = $inode, 'mode' =
$mode, 'links' = $links, 'uid' = $uid, 'gid' = $gid, 'rdev' =
$rdev, 'size' = $size, 'blksize' = $blksize, 'blocks' = $blocks,
'atime' = $atime, 'mtime' = $mtime, 'ctime' = $ctime, 'linkfi' =
$linkfi, 'flags' = $flags, 'data' = $data);
$ret = array();
foreach($encoded_values as $key = $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i strlen($val); $i++) {
$result = 6;
$result += strpos($base64, substr($val, $i , 1));
}
$ret[$key] = ($is_minus === true) ? -$result : $result;
}
return $ret;
}
and here is usage for this:
$lstat = 'gB DL+b IGg B A y A D5dZR BAA fN4 BNeR+z BNeR+7 BNeR+7 A A C';
$decoded_lstat = decode_bacula_lstat($lstat);
print_r($decoded_lstat);
Any suggestions and modifications are welcome.
Regards
gani
I tried this function. I got an interesting value for:
Bn bIgF IGk B Pp Pp BsZ/o Cp7/ZH EAA VQLA BNachF BNacW5 BNachH A A C
The PHP function returns: -1443891641
The pgsql function returns:
bacula=# select base64_decode_lstat(8,'Bn bIgF IGk B Pp Pp BsZ/o
Re: [Bacula-users] file listing?
Hi,
First, many thanks for your feedback.
I researched that behaviour with filesize of your lstat field (
-1443891641). Yes, I made mistake with using shift bitwise operator.
From PHP documentation:
Warning!
Don't right shift for more than 32 bits on 32 bits systems. Don't left
shift in case it results to number longer than 32 bits.
(source: http://php.net/manual/en/language.operators.bitwise.php)
Repairing this bug is replacing the line:
$result = 6;
to this line:
$result = bcmul($result, bcpow(2,6));
It should work properly.
Thanks.
gani
W dniu 13 marca 2011 01:01 użytkownik Dan Langille d...@langille.org napisał:
On 3/12/2011 9:29 AM, ganiuszka wrote:
W dniu 11 marca 2011 21:45 użytkownik ganiuszkaganius...@gmail.com
napisał:
Hi,
In my example jobid is putting in:
... File.JobId=8...
but this eight:
base64_decode_lstat(8,File.LStat)
is exactly eight field (encoded filesize field). I seem that you used
standard base64 decoder to decode eighth field. Am I right?
Bacula lstat is encoded by using non-standard base64 algorithm. Your
decoded lstat shows like this:
[dev] = 89
[inodes] = 1366434
[mode] = 33200
[links] = 1
[uid] = 80
[gid] = 91
[rdev] = 5481898
[size] = 315
[blksize] = 16384
[blocks] = 4
[atime] = 1299770037
[mtime] = 1299770037
[ctime] = 1299770066
[LinkFl] = 0
[flags] = 0
[data] = 2
I do not understand every field, but most fields is clear. Now, I
wrote implementation of Bacula base64_decoder in PHP. I need finish it
and make WebGUI for this and I will share this decoder here. For this
I am using source code of base64 implementation in Bacula and this:
http://old.nabble.com/The-File.LStat-field-td940366.html
Regards.
gani
2011/3/11 Dan Langilled...@langille.org:
On 3/9/2011 5:36 PM, ganiuszka wrote:
2011/3/9 Mike Egglestonmikee...@mac.com:
Afternoon,
I just noticed one of my clients had a huge incremental (level 2)
backup. I want to see what file caused the huge increase. I tried
'list
files jobid=20097' and though I'm shown the files, I'm not shown the
size of each file. Is there a command or query that shows me the size
of the file?
Mike
Hi,
Here you have SQL function for PostgreSQL (I took it from bweb):
CREATE PROCEDURAL LANGUAGE plpgsql;
BEGIN;
CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS
int8
AS $$
DECLARE
val int8;
b64 varchar(64);
size varchar(64);
i int;
BEGIN
size := split_part($2, ' ', $1);
b64 :=
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
val := 0;
FOR i IN 1..length(size) LOOP
val := val + (strpos(b64, substr(size, i, 1))-1) *
(64^(length(size)-i));
END LOOP;
RETURN val;
END;
$$ language 'plpgsql';
COMMIT;
and this is a query which lists file path, filename and size in Bytes
(in this example for jobid=8):
SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS
size FROM Filename, File, Path WHERE File.JobId=8 AND
File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER
BY size ASC;
I tested it. It works.
For the record:
In this query, I see base64_decode_lstat(8,File.LStat)
What is 8? It is not jobid. It is a field id. This lstat value:
BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C
The eight field, as defined by ' ', is E7.
--
Dan Langille - http://langille.org/
Hi,
I wrote the Bacula LStat decoder implementation in PHP. I made WebGUI
for this also. Now online LStat decoder (beta version) is available on
my page Bacula.pl:
http://www.bacula.pl/bacula-lstat-decoder/
In the near future I need to make description of fields.
Here is Bacula LStat decoder function in PHP:
function decode_bacula_lstat($lstat) {
$base64 =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
$lstat = trim($lstat);
$lstat_fields = explode(' ', $lstat);
if(count($lstat_fields) !== 16) {
die('Error! Number of lstat fields is invalid. Please make
sure that
it is valid lstat string.');
}
list($dev, $inode, $mode, $links, $uid, $gid, $rdev, $size,
$blksize,
$blocks, $atime, $mtime, $ctime, $linkfi, $flags, $data) =
$lstat_fields;
$encoded_values = array('dev' = $dev, 'inode' = $inode, 'mode' =
$mode, 'links' = $links, 'uid' = $uid, 'gid' = $gid, 'rdev' =
$rdev, 'size' = $size, 'blksize' = $blksize, 'blocks' = $blocks,
'atime' = $atime, 'mtime' = $mtime, 'ctime' = $ctime, 'linkfi' =
$linkfi, 'flags' = $flags, 'data' = $data);
$ret = array();
foreach($encoded_values as $key = $val) {
$result = 0;
$is_minus = false;
$start = 0;
if(substr($val, 0, 1) === '-') {
$is_minus = true;
$start++;
}
for($i = $start; $i strlen($val); $i++) {
$result = 6;
$result +=
Re: [Bacula-users] file listing?
On 3/9/2011 5:36 PM, ganiuszka wrote: 2011/3/9 Mike Egglestonmikee...@mac.com: Afternoon, I just noticed one of my clients had a huge incremental (level 2) backup. I want to see what file caused the huge increase. I tried 'list files jobid=20097' and though I'm shown the files, I'm not shown the size of each file. Is there a command or query that shows me the size of the file? Mike Hi, Here you have SQL function for PostgreSQL (I took it from bweb): CREATE PROCEDURAL LANGUAGE plpgsql; BEGIN; CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 AS $$ DECLARE val int8; b64 varchar(64); size varchar(64); i int; BEGIN size := split_part($2, ' ', $1); b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; val := 0; FOR i IN 1..length(size) LOOP val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); END LOOP; RETURN val; END; $$ language 'plpgsql'; COMMIT; and this is a query which lists file path, filename and size in Bytes (in this example for jobid=8): SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS size FROM Filename, File, Path WHERE File.JobId=8 AND File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER BY size ASC; I tested it. It works. For the record: In this query, I see base64_decode_lstat(8,File.LStat) What is 8? It is not jobid. It is a field id. This lstat value: BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C The eight field, as defined by ' ', is E7. -- Dan Langille - http://langille.org/ -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
Hi, In my example jobid is putting in: ... File.JobId=8... but this eight: base64_decode_lstat(8,File.LStat) is exactly eight field (encoded filesize field). I seem that you used standard base64 decoder to decode eighth field. Am I right? Bacula lstat is encoded by using non-standard base64 algorithm. Your decoded lstat shows like this: [dev] = 89 [inodes] = 1366434 [mode] = 33200 [links] = 1 [uid] = 80 [gid] = 91 [rdev] = 5481898 [size] = 315 [blksize] = 16384 [blocks] = 4 [atime] = 1299770037 [mtime] = 1299770037 [ctime] = 1299770066 [LinkFl] = 0 [flags] = 0 [data] = 2 I do not understand every field, but most fields is clear. Now, I wrote implementation of Bacula base64_decoder in PHP. I need finish it and make WebGUI for this and I will share this decoder here. For this I am using source code of base64 implementation in Bacula and this: http://old.nabble.com/The-File.LStat-field-td940366.html Regards. gani 2011/3/11 Dan Langille d...@langille.org: On 3/9/2011 5:36 PM, ganiuszka wrote: 2011/3/9 Mike Egglestonmikee...@mac.com: Afternoon, I just noticed one of my clients had a huge incremental (level 2) backup. I want to see what file caused the huge increase. I tried 'list files jobid=20097' and though I'm shown the files, I'm not shown the size of each file. Is there a command or query that shows me the size of the file? Mike Hi, Here you have SQL function for PostgreSQL (I took it from bweb): CREATE PROCEDURAL LANGUAGE plpgsql; BEGIN; CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 AS $$ DECLARE val int8; b64 varchar(64); size varchar(64); i int; BEGIN size := split_part($2, ' ', $1); b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; val := 0; FOR i IN 1..length(size) LOOP val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); END LOOP; RETURN val; END; $$ language 'plpgsql'; COMMIT; and this is a query which lists file path, filename and size in Bytes (in this example for jobid=8): SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS size FROM Filename, File, Path WHERE File.JobId=8 AND File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER BY size ASC; I tested it. It works. For the record: In this query, I see base64_decode_lstat(8,File.LStat) What is 8? It is not jobid. It is a field id. This lstat value: BZ FNmi IGw B BQ Bb U6Wq E7 EAA E BNeOq1 BNeOq1 BNeOrS A A C The eight field, as defined by ' ', is E7. -- Dan Langille - http://langille.org/ -- Większej miłości nikt nie ma nad tę, jak gdy kto życie swoje kładzie za przyjaciół swoich. Jezus Chrystus -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
Hello Mike, I'd have to say no based on the tables defined here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. The file size does not appear to be stored in the database. I don't think any size is mentioned until you get to the job level. James On 2011-03-09, at 12:51 PM, Mike Eggleston wrote: Afternoon, I just noticed one of my clients had a huge incremental (level 2) backup. I want to see what file caused the huge increase. I tried 'list files jobid=20097' and though I'm shown the files, I'm not shown the size of each file. Is there a command or query that shows me the size of the file? Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
2011/3/9 James Woodward james.woodw...@ualberta.ca: Hello Mike, I'd have to say no based on the tables defined here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. The file size does not appear to be stored in the database. I don't think any size is mentioned until you get to the job level. James I think its in the LStat. John -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On Wed, 09 Mar 2011, John Drescher might have said: 2011/3/9 James Woodward james.woodw...@ualberta.ca: Hello Mike, I'd have to say no based on the tables defined here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. The file size does not appear to be stored in the database. I don't think any size is mentioned until you get to the job level. James I think its in the LStat. John In my old version (v2.0.3) I don't see an 'lstat' command. Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
On Wed, Mar 9, 2011 at 4:52 PM, Mike Eggleston mikee...@mac.com wrote: On Wed, 09 Mar 2011, John Drescher might have said: 2011/3/9 James Woodward james.woodw...@ualberta.ca: Hello Mike, I'd have to say no based on the tables defined here http://bacula.org/5.0.x-manuals/en/developers/developers/Database_Tables.html. The file size does not appear to be stored in the database. I don't think any size is mentioned until you get to the job level. James I think its in the LStat. John In my old version (v2.0.3) I don't see an 'lstat' command. I meant that I believe the file size and attributes are in the database in the lstat column of the File table. I am not sure there is an easy way to list this. LStat tinyblobFile attributes in base64 encoding John -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] file listing?
Hi, Here you have SQL function for PostgreSQL (I took it from bweb): CREATE PROCEDURAL LANGUAGE plpgsql; BEGIN; CREATE OR REPLACE FUNCTION base64_decode_lstat(int4, varchar) RETURNS int8 AS $$ DECLARE val int8; b64 varchar(64); size varchar(64); i int; BEGIN size := split_part($2, ' ', $1); b64 := 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; val := 0; FOR i IN 1..length(size) LOOP val := val + (strpos(b64, substr(size, i, 1))-1) * (64^(length(size)-i)); END LOOP; RETURN val; END; $$ language 'plpgsql'; COMMIT; and this is a query which lists file path, filename and size in Bytes (in this example for jobid=8): SELECT Path.Path, Filename.Name, base64_decode_lstat(8,File.LStat) AS size FROM Filename, File, Path WHERE File.JobId=8 AND File.PathId=Path.PathId AND Filename.FilenameId=File.FilenameId ORDER BY size ASC; I tested it. It works. Regards gani 2011/3/9 Mike Eggleston mikee...@mac.com: Afternoon, I just noticed one of my clients had a huge incremental (level 2) backup. I want to see what file caused the huge increase. I tried 'list files jobid=20097' and though I'm shown the files, I'm not shown the size of each file. Is there a command or query that shows me the size of the file? Mike -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
