Re: [BackupPC-users] 0 repeated, 0 max chain in BackupPC 4
> > 2019-04-04 01:18:05 Cpool4 is 4118.27GB, 7700145 files (0 repeated, 0 max > chain, 6883996 max links), 17028 directories These BackupPC4 cpool stats are normal. Since BackupPC4 uses a full-file digest (unlike BackupPC3), pool collisions (different files with the same digest) are exceedingly rare, and are likely to only happen if you use known methods to create such files. Specifically, "0 repeated, 0 max chain" means there are no hash collisions. "6883996 max links" means there is some file that appears in 6883996 different places across the backups. So it is definitely finding repeated files, and a lot of them. So we need to find out why the BackupPC4 instance appears to be using a lot more resources. It's possible the BackupPC4 configuration is backing up more files than you expect (eg, sparse files, other mounted file systems?). Are you sure your exclude / include rules for each client are the same in v3 and v4? Craig On Fri, Apr 5, 2019 at 10:23 AM G.W. Haywood via BackupPC-users < backuppc-users@lists.sourceforge.net> wrote: > Hi there, > > On Fri, 5 Apr 2019, Stefan Schumacher wrote: > > > ... sorry for ... so many questions ... > > Please don't apologize for asking sensible questions. At the very > least it will help the people who try to document this stuff, and > it might even help to find some of the bugs in new software. :) > > > Old: > > 2019-03-19 06:48:27 Cpool is 1456.03GB, 8695243 files (2303 repeated, > > 373 max chain, 31999 max links), 4369 directories > > > > New: > > 2019-04-04 01:18:05 Cpool4 is 4118.27GB, 7700145 files (0 repeated, 0 > > max chain, 6883996 max links), 17028 directories > > > > Despite backing up the same servers as V3 V4 finds no repeated files > > and no chains. This is most likely the reason why the server is running > > full despite being in use less than 3 month and having triple the > > storage capacity of the old server. In V3 about 80% percent of space > > was used in the cpool, with the other 20% being allocated to pc. > > > > My questions are: Is this the cause of my capacity problems? Is this a > > problem of ZFS? How can I fix this? > > I doubt it's a ZFS problem, but I wouldn't bet money on it. > > I ran into something vaguely similar when I first started to use V4 on > systems which had been backed up using V3 for many years. Look in the > archives for my mails to the list around Sept 14-20 2018 in the thread > > BackupPC 4.2.1 apparently in an infinite loop. > > For me, the BackupPC Web interface claimed that it was using something > like three times the total storage available on the partition, which > obviously was nonsense. The issue was never entirely resolved - in > that the reason for the garbage usage figures was never identified - > but at the suggestion of someone on the list I tweaked a configuration > option and things got back to something like normal fairly quickly. > My feeling now is that V4 is still using rather more storage than V3 > (something like 70% more) to do the same job, but I haven't felt the > need to dig into it because I moved everything else off the partition > which holds the pools, and that left plenty of free space. > > I've occasionally tested recovery of a few backed-up files, there's > never been a problem. I'm not in the habit of losing large chunks of > data so I can't comment on operations more exciting than that. > > -- > > 73, > Ged. > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] 0 repeated, 0 max chain in BackupPC 4
Hi there, On Fri, 5 Apr 2019, Stefan Schumacher wrote: ... sorry for ... so many questions ... Please don't apologize for asking sensible questions. At the very least it will help the people who try to document this stuff, and it might even help to find some of the bugs in new software. :) Old: 2019-03-19 06:48:27 Cpool is 1456.03GB, 8695243 files (2303 repeated, 373 max chain, 31999 max links), 4369 directories New: 2019-04-04 01:18:05 Cpool4 is 4118.27GB, 7700145 files (0 repeated, 0 max chain, 6883996 max links), 17028 directories Despite backing up the same servers as V3 V4 finds no repeated files and no chains. This is most likely the reason why the server is running full despite being in use less than 3 month and having triple the storage capacity of the old server. In V3 about 80% percent of space was used in the cpool, with the other 20% being allocated to pc. My questions are: Is this the cause of my capacity problems? Is this a problem of ZFS? How can I fix this? I doubt it's a ZFS problem, but I wouldn't bet money on it. I ran into something vaguely similar when I first started to use V4 on systems which had been backed up using V3 for many years. Look in the archives for my mails to the list around Sept 14-20 2018 in the thread BackupPC 4.2.1 apparently in an infinite loop. For me, the BackupPC Web interface claimed that it was using something like three times the total storage available on the partition, which obviously was nonsense. The issue was never entirely resolved - in that the reason for the garbage usage figures was never identified - but at the suggestion of someone on the list I tweaked a configuration option and things got back to something like normal fairly quickly. My feeling now is that V4 is still using rather more storage than V3 (something like 70% more) to do the same job, but I haven't felt the need to dig into it because I moved everything else off the partition which holds the pools, and that left plenty of free space. I've occasionally tested recovery of a few backed-up files, there's never been a problem. I'm not in the habit of losing large chunks of data so I can't comment on operations more exciting than that. -- 73, Ged. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] 0 repeated, 0 max chain in BackupPC 4
Hello, I am sorry for "flooding" the mailing-list with so many questions at once but I am currently in the process of debugging a problem and as part of that process new questions have arisen that I could not yet formulate at the time of my previous postings. I have two backup servers. The old server uses BackupPC 3 and ext4, the new server, which is causing me problems, uses BackupPC 4 and zfs. While comparing logs I have found the two entries which I believe could point in the direction of the problem. Old: 2019-03-19 06:48:27 Cpool is 1456.03GB, 8695243 files (2303 repeated, 373 max chain, 31999 max links), 4369 directories New: 2019-04-04 01:18:05 Cpool4 is 4118.27GB, 7700145 files (0 repeated, 0 max chain, 6883996 max links), 17028 directories Despite backing up the same servers as V3 V4 finds no repeated files and no chains. This is most likely the reason why the server is running full despite being in use less than 3 month and having triple the storage capacity of the old server. In V3 about 80% percent of space was used in the cpool, with the other 20% being allocated to pc. My questions are: Is this the cause of my capacity problems? Is this a problem of ZFS? How can I fix this? Yours sincerely Stefan Stefan Schumacher Systemadministrator NetFederation GmbH Sürther Hauptstraße 180 B - Fon:+49 (0)2236/3936-701 E-Mail: stefan.schumac...@net-federation.de Internet: http://www.net-federation.de Besuchen Sie uns doch auch auf facebook, twitter, Google+, flickr, Slideshare, XING oder unserem Blog. Wir freuen uns! *** HR Benchmark 2019: Bewerber in den Mittelpunkt! *** Wie bewerberfreundlich und auf welchem digitalen Stand sind die Karriereportale großer deutscher Unternehmen? Unsere Antworten finden Sie in unserem frisch veröffentlichten HR Benchmark unter: https://www.hr-benchmark.de/human-resources-benchmark-2019/benchmark * NetFederation GmbH Geschäftsführung: Christian Berens, Thorsten Greiten Amtsgericht Köln, HRB Nr. 32660 * The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee and access to the e-mail by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this e-mail in error please forward to: post...@net-federation.de Die in dieser E-Mail enthaltenen Informationen sind vertraulich und können von rechtlicher Bedeutung sein. Diese Mail ist ausschließlich für den Adressaten bestimmt und jeglicher Zugriff durch andere Personen ist nicht zulässig. Falls Sie nicht der beabsichtigte Empfänger sind, ist jegliche Veröffentlichung, Vervielfältigung, Verteilung oder sonstige in diesem Zusammenhang stehende Handlung untersagt und unter Umständen ungesetzlich. Falls Sie diese E-Mail irrtümlich erhalten haben, leiten Sie sie bitte weiter an: post...@net-federation.de ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
