Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
On Fri, Sep 17, 2010 at 07:15:49PM +0800, Chris Hoy Poy wrote: > Hi Robin, > >- disk fragmentation coupled with minimum file size allocation >issues (?) ie lots of the same large file updated in place >might lead to this scenario, I think? I guess? Don't know how I'd check that. >- can you confirm trash is empty? It is empty. > - is nightly cleanup showing that it actually cleaned stuff up? > (I have "Nightly cleanup removed 31159 files of size 88.77GB > (around 9/17 15:37)" or similiar on status) Nightly cleanup removed 2831 files of size 28.75GB (around 9/19 02:39), >- how many full backups / incremental backups are listed on >your "host summary" page ? There are 172 hosts that have been backed up, for a total of: 318 full backups of total size 2569.41GB (prior to pooling and compression), 4687 incr backups of total size 8298.06GB (prior to pooling and compression). > My pool is around 630GB and 3843035 files (so 1/5th the size and > only half the number of files), but DF shows it to be 654GB on > disk. That's a 2% difference; min is 3055GiB vs ~3800GiB, which is a *20*% difference. 10x more different. Seems a lot. > I put the discrepancy down to the number of directories in my PC > tree - directories don't hardlink, only files do. The list of > files in a directory has to be stored somewhere, and each > directory has a minimum allocation size on disk. Good point. That might do it. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
On Friday 17 Sep 2010 12:15:49 Chris Hoy Poy wrote: >My pool is around 630GB and 3843035 files (so 1/5th the size and only > half the number of files), but DF shows it to be 654GB on disk. The first question we should as is "Does BackupPC's pool usage report actual file size, or file size on disk (rounded up to nearest block)?" Does anyone know the answer? I would expect it to show actual file size, because "size on disk" will vary with filesystem, both at the BackupPC server and the host you might restore it to. Only a true file size report would be useful. But it would be nice to report and graph both. Regards, Tyler -- "Religion is autocratic, law is democratic, and science is meritocratic. Religious logic is dogmatic, legal logic is axiomatic, and science is pragmatic. Scientific theories can be altered by publishing a paper with reproducible results, and political principles can be changed every two to four years with an election, but if you want to change religious principles you usually have to wait for a whole generation of clergy to die." -- Soren Ragsdale, http://tongodeon.livejournal.com/801252.html -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
Wasn't there a gotcha' running the nightly cleanup manually? -- /Sorin >-Original Message- >From: Chris Hoy Poy [mailto:krya...@gopc.net] >Sent: Friday, September 17, 2010 1:16 PM >To: General list for user discussion, questions and support >Subject: Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't >think) > >Hi Robin, > > - is nightly cleanup showing that it actually cleaned stuff up? (I have "Nightly >cleanup removed 31159 files of size 88.77GB (around 9/17 15:37)" or similiar on >status) smime.p7s Description: S/MIME cryptographic signature -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think)
Hi Robin, - disk fragmentation coupled with minimum file size allocation issues (?) ie lots of the same large file updated in place might lead to this scenario, I think? - can you confirm trash is empty? is the trash cleanup daemon getting stuck and therefore nothing getting cleaned up? - is nightly cleanup showing that it actually cleaned stuff up? (I have "Nightly cleanup removed 31159 files of size 88.77GB (around 9/17 15:37)" or similiar on status) - how many full backups / incremental backups are listed on your "host summary" page ? My pool is around 630GB and 3843035 files (so 1/5th the size and only half the number of files), but DF shows it to be 654GB on disk. I put the discrepancy down to the number of directories in my PC tree - directories don't hardlink, only files do. The list of files in a directory has to be stored somewhere, and each directory has a minimum allocation size on disk. //chris - Original Message - From: "Robin Lee Powell" To: "BackupPC Users" Sent: Friday, 17 September, 2010 1:07:25 AM Subject: [BackupPC-users] Surprising disk usage, *not* a link problem (I don't think) I have a fairly large (171 hosts) backup environment that seems to be using rather more disk than it should. GUI says: Pool is 3055.59GB comprising 7361233 files and 4369 directories (as of 9/16 01:33), df says: Filesystem Size Used Avail Use% Mounted on /dev/mapper/local-backups 4.0T 3.8T 270G 94% /backups which is a pretty significant gap. I've never moved anything around, so it shouldn't be a linking problem. The trash is empty. Running "du -h --exclude='[0-9]*' --exclude=new" in the pc directory, which *should* ignore only the directories that link to the pool, gives "2.4G ." Running a proper du of the whole tree would take A While, and I'm not sure my RAM could survive the link checking along with all the other stuff backuppc is doing. Any ideas as to what's going on here? Maybe relevant: $Conf{BackupPCNightlyPeriod} = 8; But that still seems excessive? I've run the nightly manually a few times, as "sudo -u backuppc /usr/local/bin/BackupPC_nightly 0 255" (is that the correct syntax for doing that?) and it's taken well over 24 hours, so splitting it up seems like a probably good idea? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/