Hello,
my perl knowledge is very minimal, so I am unable to use poolCnt files
until someone will describe them as documentation.
But I used BackupPC_ls to get needed information. Counted only files
("-" in attrs) it's size divided by number of links, which makes me best
results to compare, which backup is largest.
If someone needs this, my python script live here:
http://www.salstar.sk/pub/salpack/usr/bin/backuppc_size.py
SAL
On Wed, Sep 02, 2020 at 11:12:48PM -0700, Craig Barratt wrote:
> As I said in my previous email, I'd recommend using the poolCnt files since
> they accurately reflect what is being stored for each host. But it will
> take a bit more coding. You can look in some of the BackupPC utilities
> (eg: BackupPC_refCountUpdate) for examples of how to read the poolCnt files.
>
> Craig
>
> On Wed, Sep 2, 2020 at 11:03 PM Ján ONDREJ (SAL) <ondr...@salstar.sk> wrote:
>
> > Hello,
> >
> > trying to make a script, which can count backup size, but without success
> > yet. I can parse XferLOG, extract sizes, identify links, but after summary
> > some backups have zero or almost zero size. Looks like this happens, when
> > there is no initial backup (backup with id 0), where all files have been
> > transferred. My XferLOGs contains only changes.
> >
> > Here is what I counted from XferLOG for one server:
> > digitall/XferLOG.625.z: 4.788 MB
> > digitall/XferLOG.628.z: 0.899 MB
> > digitall/XferLOG.629.z: 19.221 MB
> > digitall/XferLOG.623.z: 0.059 MB
> > digitall/XferLOG.627.z: 0.000 MB
> > digitall/XferLOG.622.z: 0.065 MB
> > digitall/XferLOG.624.z: 0.138 MB
> > digitall/XferLOG.626.z: 0.060 MB
> > HOST: digitall: 25.231 MB
> >
> > As you see, all files from XferLOG uses 25 MB of disk space.
> > But I see an 4 GB file stored in my backup, when checking from BackupPC web
> > interface.
> >
> > I still think, that this information about backup File Size is useless.
> > What does it mean?
> > 1. how much files have been transferred: NO
> > 2. how much files are stored on disk: NO
> > 3. how much files are on source filesystem: NO
> >
> > I really don't know, what this means.
> >
> > Please, at least let me know, how I can check, how much files are stored
> > in each backup. I can parse them all using BackupPC_tarCreate or something
> > similar, or using backuppcfs, but this is very slow for large backups.
> >
> > SAL
> >
> > On Mon, Aug 31, 2020 at 12:20:13PM -0700, Craig Barratt wrote:
> > > I've tried to suggest a couple of reasons that could explain what you are
> > > seeing, based on very incomplete information.
> > >
> > > Without you confirming what the issue actually is, your conclusion is
> > > already that it's "absolutely buggy" and "useless". To pick another
> > > example, rsync -aHv will also report a total file size that is the sum of
> > > the hardlink file sizes, and it also reports the actual bytes transferred
> > > and the speedup. So by your logic, does that mean rsync is also
> > > "absolutely buggy" and "useless"?
> > >
> > > I'd recommend you actually understand the issue, and then decide what the
> > > best options are.
> > >
> > > If your question is "is there a reasonable way to potion pool usage among
> > > backup clients?" you are asking a question that doesn't have a simple
> > > answer, because of hardlinks on the clients and pooling among all the
> > > clients.
> > >
> > > That said, it wouldn't be too hard to write a script that reads the
> > > reference counts for a client (which includes all the backups for that
> > > client), and apportioning the pool file sizes to that client based on the
> > > ratio of its own reference count to the total pool reference count for
> > each
> > > pool file. But that's just one way of doing it. And commercial
> > > auditing/billing tools are well out of scope for BackupPC, but you are
> > most
> > > welcome to contribute anything you develop.
> > >
> > > Craig
> > >
> > > On Mon, Aug 31, 2020 at 9:38 AM Ján ONDREJ (SAL) <ondr...@salstar.sk>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > thanks for explanation, but how I can check in backuppc, which user
> > > > uses how much disk space of my backuppc storage? This way File Size
> > counter
> > > > is absolutelly buggy.
> > > >
> > > > I need to check, which backup uses most of my space and need to find,
> > > > where I should exclude more files. But there is no information, which
> > > > I can use. New files are only new files, doesn't count how much files
> > > > there are. In Total files hardlinked files are counter multiple times,
> > > > which ends in 10x more space usage in this counter like it's real.
> > > >
> > > > This way total files counter is useless, only useful for windows
> > users,
> > > > which don't use hardlinks.
> > > >
> > > > SAL
> > > >
> > > > On Mon, Aug 31, 2020 at 09:09:02AM -0700, Craig Barratt via
> > BackupPC-users
> > > > wrote:
> > > > > That file is a hardlink, not a symlink. In the backup stats, each
> > > > instance
> > > > > of a hardlink is counted towards the total file size.
> > > > >
> > > > > If your file system has a lot of hardlinks, perhaps that's why the
> > > > reported
> > > > > number is higher than you expect?
> > > > >
> > > > > Craig
> > > > >
> > > > > On Mon, Aug 31, 2020 at 12:40 AM Ján ONDREJ (SAL) <
> > ondr...@salstar.sk>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Mon, Aug 31, 2020 at 12:08:50AM -0700, Craig Barratt via
> > > > BackupPC-users
> > > > > > wrote:
> > > > > > > Does your backup include sparse files?
> > > > > >
> > > > > > I think no.
> > > > > >
> > > > > > > Look in the XferLOG file to see the sizes of individual files -
> > it
> > > > > > > shouldn't be too hard to spot one that is large.
> > > > > >
> > > > > > There is no one large file. As I wrote, restored backup is not so
> > large
> > > > > > too.
> > > > > > But you pointed me to right plate. I see this line in XferLOG:
> > > > > >
> > > > > > new recv hf..tpog... rw-r--r-- 1000, 1000 25089367
> > > > > > var/www/public/media/598522/catalogue.pdf =>
> > > > > > var/www/private/import/docs/catalogue.pdf
> > > > > >
> > > > > > This is a symlink and it's size is counted as 25089367.
> > > > > > According to "=>" symbol, this symlink is properly identified as
> > > > symlink,
> > > > > > but it's size is stored as symlinks target file. This is why backup
> > > > > > size is larger than my filesystem. Can this be fixed?
> > > > > >
> > > > > > SAL
> > > > > >
> > > > > > > On Sun, Aug 30, 2020 at 11:51 PM Ján ONDREJ (SAL) <
> > > > ondr...@salstar.sk>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I am using BackupPC for years, but after update to v4
> > (4.4.0),
> > > > some
> > > > > > > > backups have inconsistent size dislayed in "File Size/Count
> > Reuse
> > > > > > Summary"
> > > > > > > > table.
> > > > > > > >
> > > > > > > > This is my servers directory, which should be in backup:
> > > > > > > >
> > > > > > > > Filesystem Size Used Avail Use% Mounted on
> > > > > > > > /dev/md0 4.0G 3.0G 1.1G 75% /
> > > > > > > > /dev/mapper/vg_server-www 200G 119G 82G 60% /var/www
> > > > > > > >
> > > > > > > > There is no other filesystem mounted, binded to this directory.
> > > > > > > >
> > > > > > > > It's complete size is 200 GB, used only 118 GB. Some files are
> > > > excluded
> > > > > > > > from
> > > > > > > > backup, so an full backup should be about 120 GB uncompressed,
> > > > after
> > > > > > > > compression less. But our File Size table looks like:
> > > > > > > >
> > > > > > > > Totals Existing Files New
> > > > Files
> > > > > > > > Backup# Type #Files Size/MiB MiB/sec #Files Size/MiB
> > #Files
> > > > > > Size/MiB
> > > > > > > > 0 full 3220584 943488.6 26.86 2625461 872577.7 1232
> > > > 537.4
> > > > > > > >
> > > > > > > > As you see, this backup total size is 921 GiB. How it's
> > possible,
> > > > that
> > > > > > > > an 200 GB partition is stored as 900 GB?
> > > > > > > > Also according to i-nodes in linux, my server has:
> > > > > > > >
> > > > > > > > Filesystem Inodes IUsed IFree IUse%
> > > > Mounted on
> > > > > > > > /dev/md0 4194240 51082 4143158 2% /
> > > > > > > > /dev/mapper/vg_fusion-www 209715200 2470434 207244766 2%
> > > > /var/www
> > > > > > > >
> > > > > > > > So there is 2.5 millions of files, some excluded, but results
> > as
> > > > 3.2
> > > > > > > > millions
> > > > > > > > on backup.
> > > > > > > >
> > > > > > > > Trying to restore files. Restore downloaded a 68GB tar package.
> > > > > > > > This looks to be real, but if there are only 70 GB of data,
> > why on
> > > > > > backuppc
> > > > > > > > status it's displayed as 900 GB?
> > > > > > > >
> > > > > > > > I need to find, which server is using most of my backup space.
> > > > > > > > I know, that it's hard to find, because files are shared
> > between
> > > > > > servers
> > > > > > > > (deduplicated), but at least I can estimate it. But if there
> > are
> > > > > > multiples
> > > > > > > > of usage displayed in statistics, then it's impossible to
> > > > aproximate.
> > > > > > > >
> > > > > > > > I deleted all backups of this server, in hope, that it helps,
> > but
> > > > > > don't.
> > > > > > > > :-(
> > > > > > > >
> > > > > > > > Thank for help.
> > > > > > > >
> > > > > > > > SAL
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > BackupPC-users mailing list
> > > > > > > > BackupPC-users@lists.sourceforge.net
> > > > > > > > List:
> > > > https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > > > > Wiki: https://github.com/backuppc/backuppc/wiki
> > > > > > > > Project: https://backuppc.github.io/backuppc/
> > > > > > > >
> > > > > >
> > > > > >
> > > > > > > _______________________________________________
> > > > > > > BackupPC-users mailing list
> > > > > > > BackupPC-users@lists.sourceforge.net
> > > > > > > List:
> > https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > > > Wiki: https://github.com/backuppc/backuppc/wiki
> > > > > > > Project: https://backuppc.github.io/backuppc/
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > BackupPC-users mailing list
> > > > > > BackupPC-users@lists.sourceforge.net
> > > > > > List:
> > https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > > Wiki: https://github.com/backuppc/backuppc/wiki
> > > > > > Project: https://backuppc.github.io/backuppc/
> > > > > >
> > > >
> > > >
> > > > > _______________________________________________
> > > > > BackupPC-users mailing list
> > > > > BackupPC-users@lists.sourceforge.net
> > > > > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > Wiki: https://github.com/backuppc/backuppc/wiki
> > > > > Project: https://backuppc.github.io/backuppc/
> > > >
> > > >
> >
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
