Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
On Fri, Sep 17, 2010 at 07:15:49PM +0800, Chris Hoy Poy wrote: Hi Robin, - disk fragmentation coupled with minimum file size allocation issues (?) ie lots of the same large file updated in place might lead to this scenario, I think? I guess? Don't know how I'd check that. - can you confirm trash is empty? It is empty. - is nightly cleanup showing that it actually cleaned stuff up? (I have Nightly cleanup removed 31159 files of size 88.77GB (around 9/17 15:37) or similiar on status) Nightly cleanup removed 2831 files of size 28.75GB (around 9/19 02:39), - how many full backups / incremental backups are listed on your host summary page ? There are 172 hosts that have been backed up, for a total of: 318 full backups of total size 2569.41GB (prior to pooling and compression), 4687 incr backups of total size 8298.06GB (prior to pooling and compression). My pool is around 630GB and 3843035 files (so 1/5th the size and only half the number of files), but DF shows it to be 654GB on disk. That's a 2% difference; min is 3055GiB vs ~3800GiB, which is a *20*% difference. 10x more different. Seems a lot. I put the discrepancy down to the number of directories in my PC tree - directories don't hardlink, only files do. The list of files in a directory has to be stored somewhere, and each directory has a minimum allocation size on disk. Good point. That might do it. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
Hi Robin,
- disk fragmentation coupled with minimum file size allocation issues (?) ie
lots of the same large file updated in place might lead to this scenario, I
think?
- can you confirm trash is empty? is the trash cleanup daemon getting stuck
and therefore nothing getting cleaned up?
- is nightly cleanup showing that it actually cleaned stuff up? (I have
Nightly cleanup removed 31159 files of size 88.77GB (around 9/17 15:37) or
similiar on status)
- how many full backups / incremental backups are listed on your host
summary page ?
My pool is around 630GB and 3843035 files (so 1/5th the size and only half
the number of files), but DF shows it to be 654GB on disk. I put the
discrepancy down to the number of directories in my PC tree - directories don't
hardlink, only files do. The list of files in a directory has to be stored
somewhere, and each directory has a minimum allocation size on disk.
//chris
- Original Message -
From: Robin Lee Powell rlpow...@digitalkingdom.org
To: BackupPC Users backuppc-users@lists.sourceforge.net
Sent: Friday, 17 September, 2010 1:07:25 AM
Subject: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't
think)
I have a fairly large (171 hosts) backup environment that seems to
be using rather more disk than it should.
GUI says: Pool is 3055.59GB comprising 7361233 files and 4369
directories (as of 9/16 01:33),
df says:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/local-backups 4.0T 3.8T 270G 94% /backups
which is a pretty significant gap. I've never moved anything
around, so it shouldn't be a linking problem.
The trash is empty.
Running du -h --exclude='[0-9]*' --exclude=new in the pc
directory, which *should* ignore only the directories that link to
the pool, gives 2.4G .
Running a proper du of the whole tree would take A While, and I'm
not sure my RAM could survive the link checking along with all the
other stuff backuppc is doing.
Any ideas as to what's going on here?
Maybe relevant:
$Conf{BackupPCNightlyPeriod} = 8;
But that still seems excessive? I've run the nightly manually a few
times, as sudo -u backuppc /usr/local/bin/BackupPC_nightly 0 255
(is that the correct syntax for doing that?) and it's taken well
over 24 hours, so splitting it up seems like a probably good idea?
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which this parrot
is dead is ti poi spitaki cu morsi, but this sentence is false
is na nei. My personal page: http://www.digitalkingdom.org/rlp/
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
___ BackupPC-users mailing
list BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
Wasn't there a gotcha' running the nightly cleanup manually? -- /Sorin -Original Message- From: Chris Hoy Poy [mailto:krya...@gopc.net] Sent: Friday, September 17, 2010 1:16 PM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think) Hi Robin, - is nightly cleanup showing that it actually cleaned stuff up? (I have Nightly cleanup removed 31159 files of size 88.77GB (around 9/17 15:37) or similiar on status) smime.p7s Description: S/MIME cryptographic signature -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
On Friday 17 Sep 2010 12:15:49 Chris Hoy Poy wrote: My pool is around 630GB and 3843035 files (so 1/5th the size and only half the number of files), but DF shows it to be 654GB on disk. The first question we should as is Does BackupPC's pool usage report actual file size, or file size on disk (rounded up to nearest block)? Does anyone know the answer? I would expect it to show actual file size, because size on disk will vary with filesystem, both at the BackupPC server and the host you might restore it to. Only a true file size report would be useful. But it would be nice to report and graph both. Regards, Tyler -- Religion is autocratic, law is democratic, and science is meritocratic. Religious logic is dogmatic, legal logic is axiomatic, and science is pragmatic. Scientific theories can be altered by publishing a paper with reproducible results, and political principles can be changed every two to four years with an election, but if you want to change religious principles you usually have to wait for a whole generation of clergy to die. -- Soren Ragsdale, http://tongodeon.livejournal.com/801252.html -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
