Re: [BackupPC-users] Can't access older backups after upgrade
On 2021-11-04 17:00, Anthony Chavez wrote: On 11/4/21 3:09 PM, Norman Goldstein wrote: On 2021-11-04 07:36, Les Mikesell wrote: On Thu, Nov 4, 2021 at 4:16 AM Anthony Chavez wrote: Namely? I use NFS for a lot of things. And apart from an abysmal security model (that can fortunately be hardened with a lot of extra work), I've had no complaints. Think about the network activity that has to happen when you rsync a file with small changes when it is stored over an NFS mount. The whole thing has to be read over the network to find the differences - when most of the point of rsync is to avoid that network traffic. That's not your immediate problem but something to consider when putting the backuppc pool on NFS. Which NAS OS'es/protocols have worked best with backupc, in your experience? I have been using NFS on a NAS for years, as the pool location for backuppc. It has had its ups and downs. Seems to me that Les is talking about simply making good use of bandwidth here. I think the point he is making is that ANY network filesystem would effectively double (or worse) the amount of traffic generated by rsync. And it seems obvious that ideally, the storage pool should reside on the same machine as the one running BackupPC and NFS, CIFS, DFS, gluster, ceph, or any other network filesystem should be avoided in general. HOWEVER, what I neglected to mention in my initial post is that in MY case, the NFS daemon is running on the host whereas the NFS client is running in a VM *on*the*same*hardware.* So in my case, the performance hit should have been more or less negligible, especially because this hardware is dedicated to backups only. Unfortunately, I'm not sure bhyve (FreeBSD's hypervisor) supports virtio-9p yet so NFS is necessary in order to mount an external ZFS dataset inside a VM. Last I checked into it, support was "just around the corner" but regardless, TrueNAS's UI does not provide access to that functionality. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project:https://backuppc.github.io/backuppc/ I see. Thanks for the explanation. Bandwidth has not been a problem, yet, but I will keep that in mind. I like having the backup isolated on separate hardware. It seems it makes most sense to have the pool on the disk of the backuppc server when the server is dedicated to backups i.e. it behaves as a fancy NAS that can run backuppc server :-)___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
Regarding my situation, here's the short version at this point: I've metadata but no data. I've no idea how it happened, but something obviously did, somewhere in here: oldhost% zfs send | ssh backuphost zfs recv backuphost% zfs send | ssh newhost zfs recv newhost% zfs list -t snapshot | xargs zfs destroy I was replacing old hardware with new hardware and none of these commands are exact, but get the idea across. I've no *conclusive* proof, but it seems to me that command #3 above is where things went wrong. BackupPC is not to blame, which is great because I love it and want to keep using it. Lesson learned: don't destroy snapshots until you've validated your data. Duh. While these backups ARE valuable (to me alone), fortunately, most of the data CAN be restored but it's going to take a lot more effort than I would like. C'est la vie. On 11/2/21 8:34 AM, backu...@kosowsky.org wrote: Have you tried BackupPC_zcat on the above pool file? BackupPC_zcat: can't find /data/backuppc/cpool/72/ca/72ca0cc1bf7a179f8c3cdabc187e3887 What is in the refCnt directory (and subdirectory)? What is in the f-mangled directories? What is in the log files under: - TopDir/log - TopDir/pc/$host Are there errors? And if so what are they? I can provide all the above if you still would like, but it seems to me that something has gone horribly wrong and is causing my backups to be quickly eroded away. This is from the Web UI: Nightly cleanup removed 188102 files of size 182.78GiB (around 2021-11-03 01:42), As G.W. pointed out, there are some unattractive "steps" in week 44. I concur, the pool appears to be shrinking due to some sort of configuration corruption. I think it's pretty clear that it's a lost cause at this point and I will need to start over from scratch. You may also want to try backuppcfs.pl which mounts your backups as a fuse file system and allows you to browse files to see what if any files are actually backed up. Is this part of the standard distribution tarball? It doesn't seem to be installed and I don't see it on GitHub. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On 11/4/21 3:09 PM, Norman Goldstein wrote: On 2021-11-04 07:36, Les Mikesell wrote: On Thu, Nov 4, 2021 at 4:16 AM Anthony Chavez wrote: Namely? I use NFS for a lot of things. And apart from an abysmal security model (that can fortunately be hardened with a lot of extra work), I've had no complaints. Think about the network activity that has to happen when you rsync a file with small changes when it is stored over an NFS mount. The whole thing has to be read over the network to find the differences - when most of the point of rsync is to avoid that network traffic. That's not your immediate problem but something to consider when putting the backuppc pool on NFS. Which NAS OS'es/protocols have worked best with backupc, in your experience? I have been using NFS on a NAS for years, as the pool location for backuppc. It has had its ups and downs. Seems to me that Les is talking about simply making good use of bandwidth here. I think the point he is making is that ANY network filesystem would effectively double (or worse) the amount of traffic generated by rsync. And it seems obvious that ideally, the storage pool should reside on the same machine as the one running BackupPC and NFS, CIFS, DFS, gluster, ceph, or any other network filesystem should be avoided in general. HOWEVER, what I neglected to mention in my initial post is that in MY case, the NFS daemon is running on the host whereas the NFS client is running in a VM *on*the*same*hardware.* So in my case, the performance hit should have been more or less negligible, especially because this hardware is dedicated to backups only. Unfortunately, I'm not sure bhyve (FreeBSD's hypervisor) supports virtio-9p yet so NFS is necessary in order to mount an external ZFS dataset inside a VM. Last I checked into it, support was "just around the corner" but regardless, TrueNAS's UI does not provide access to that functionality. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On 2021-11-04 07:36, Les Mikesell wrote: On Thu, Nov 4, 2021 at 4:16 AM Anthony Chavez wrote: Namely? I use NFS for a lot of things. And apart from an abysmal security model (that can fortunately be hardened with a lot of extra work), I've had no complaints. Think about the network activity that has to happen when you rsync a file with small changes when it is stored over an NFS mount. The whole thing has to be read over the network to find the differences - when most of the point of rsync is to avoid that network traffic. That's not your immediate problem but something to consider when putting the backuppc pool on NFS. Which NAS OS'es/protocols have worked best with backupc, in your experience? I have been using NFS on a NAS for years, as the pool location for backuppc. It has had its ups and downs. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On 11/4/21 7:29 AM, jbk wrote: It appears to me that backuppc is no longer owner of the older backups because you changed distro's and the uid/gid has changed for the backuppc user. Your going to have to run chown backkuppc.backuppc on all the older folders & files to allow backuppc to see them. This takes a long time to complete, at least it did for me. Jimkr Hi Jim, Thanks for your input. chown was one of the first things I did. In fact I was verifying exactly that when I posted the output of ls -n to the mailing list. Here is the same directory without -n, from inside the Docker container. As you can see, it's up to snuff but BackupPC nonetheless thinks the directory is empty. $ ls -l /data/backuppc/pc/$host/$baknum total 152 -rw-r- 1 backuppc backuppc 0 Sep 20 23:17 attrib_94b54db94ac9ff868491c3b189178d62 -rw-r- 1 backuppc backuppc 665 Sep 20 23:24 backupInfo drwxr-x--- 2 backuppc backuppc 3 Sep 20 19:00 f%2fboot drwxr-x--- 3 backuppc backuppc 4 Sep 20 19:01 f%2fesp drwxr-x--- 144 backuppc backuppc 145 Sep 20 20:02 f%2fetc drwxr-x--- 3 backuppc backuppc 4 Sep 20 20:03 f%2fhome drwxr-x--- 4 backuppc backuppc 5 Sep 20 21:54 f%2fopt drwxr-x--- 13 backuppc backuppc 14 Sep 20 21:56 f%2froot drwxr-x--- 5 backuppc backuppc 6 Sep 20 22:07 f%2fsrv drwxr-x--- 11 backuppc backuppc 12 Sep 20 23:11 f%2fusr%2flocal drwxr-x--- 5 backuppc backuppc 6 Sep 20 23:17 f%2fvar drwxr-x--- 8 backuppc backuppc 9 Sep 20 23:18 f%2fvar%2flib drwxr-x--- 7 backuppc backuppc 7 Sep 20 23:11 inode drwxr-x--- 2 backuppc backuppc 130 Oct 7 14:19 refCnt ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On 11/2/21 05:08, Anthony Chavez wrote: On 10/19/21 7:06 PM, Anthony Chavez wrote: On 10/9/21 3:02 PM, Anthony Chavez wrote: Hi all, So I have had to make a slight change to my BackupPC setup and while I am able to access new backups in the web interface, I am now unable to access my older backups. My old setup was under a FreeNAS Jail, using the FreeBSD pkg of BackupPC with a nullfs mount from the jail to a ZFS dataset housing the backups themselves. It worked great for months. When upgrading from FreeNAS 11 to TrueNAS 12, my jail refused to upgrade so the option was to either create a new jail or install Linux in a VM and mount the backups directory over NFSv3. I chose the latter (ArchLinux) and because Docker is becoming a thing in my LAN, adferrand/backuppc. The error message for older backups I am seeing in the web interface is "Error: Directory /data/backuppc/pc/$host/$baknum is empty". It appears to me that backuppc is no longer owner of the older backups because you changed distro's and the uid/gid has changed for the backuppc user. Your going to have to run chown backkuppc.backuppc on all the older folders & files to allow backuppc to see them. This takes a long time to complete, at least it did for me. Jimkr When trying to access an older backup, BackupPC_ls comes back with, for example: "bpc_attrib_dirRead: can't open /data/backuppc/cpool/72/ca/72ca0cc1bf7a179f8c3cdabc187e3887". Permissions check out (uid 126/backuppc, gid 126/backuppc, umask 027) so the only thing I can think of is that I need to re-index or re-build the pool somehow, but I am not familiar enough with BackupPC command line utilities to know what to do at this point. Any help would be much appreciated. Could anyone help with this please? It's making no sense to me: $ ls -ahln /data/backuppc/pc/$host/$baknum total 65K drwxr-x--- 8 126 126 10 Aug 29 16:22 . drwxr-x--- 110 126 126 233 Oct 19 15:12 .. -rwxr-x--- 1 126 126 0 Aug 29 16:22 attrib_5e6a2266a222d07587f687f9df5ac097 -rwxr-x--- 1 126 126 695 Aug 29 16:00 backupInfo drwxr-x--- 3 126 126 4 Aug 29 16:01 f%2fetc drwxr-x--- 3 126 126 4 Aug 29 16:14 f%2fhome drwxr-x--- 3 126 126 3 Aug 29 16:18 f%2fsrv drwxr-x--- 4 126 126 4 Aug 29 16:20 f%2fvar drwxr-x--- 2 126 126 3 Aug 29 16:22 f%2fvar%2flib drwxr-x--- 2 126 126 95 Oct 7 14:13 refCnt $ /usr/local/BackupPC/bin/BackupPC_ls -h $host -n $baknum -s /etc / /: bpc_attrib_dirRead: can't open /data/backuppc/cpool/72/ca/72ca0cc1bf7a179f8c3cdabc187e3887 bpc_attrib_dirRead: can't open /data/backuppc/cpool/68/40/6940aeccff660743da3041191216d260 Is there nobody that can help me with this? Are my backups lost? I am open to paying for one-on-one assistance with this. I would very much like to salvage this situation, if at all possible. -- Jim Kelly-Rand j...@kjkelra.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On Thu, Nov 4, 2021 at 4:16 AM Anthony Chavez wrote: > > Namely? > > I use NFS for a lot of things. And apart from an abysmal security model (that > can fortunately be hardened with a lot of extra work), I've had no complaints. Think about the network activity that has to happen when you rsync a file with small changes when it is stored over an NFS mount. The whole thing has to be read over the network to find the differences - when most of the point of rsync is to avoid that network traffic. That's not your immediate problem but something to consider when putting the backuppc pool on NFS. -- Les Mikesell lesmikes...@gmail.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
Hi there, On Thu, 4 Nov 2021, Anthony Chavez wrote: On 11/2/21 7:54 AM, G.W. Haywood via BackupPC-users wrote: Hi G.W.! Thanks for responding. :) FWIW I wouldn't use NFS for anything that even vaguely bears a passing resemblance to a backup system.? In my experience NFS is always flaky. Even if you do get it more or less under control, there always seem to be gotchas lurking in there. Namely? Well my absolute favourite was when the OOM killer decided that nfsd was the obvious candidate. A *lot* of machines went down that day. ... apart from an abysmal security model ... There you go! Indirection of configuration variables? I'm not so sure that's an issue with my config, but how would I know? There are some values like TOPDIR, you could check them. what you have done so far seems to me a bit on the rash side. How do you mean? You had what appears to have been a working backup system. Then you changed it, and it no longer works. So I would like to think that point (2) you make is inaccurate and would hope that in doing so my backups are indeed safe. Well, perhaps I could have phrased it better. :) But you do seem to have broken it. When I decided on upgrading to version 4.x of BPC, I left one machine which ran version 3 *completely* untouched and installed a *completely* new machine. So then I had three backups running nightly. Yes, there were two already. After I got BPC4 in some semblance of order I turned off one of the machines (but it's still sitting there, turned off, and BPC says it now hasn't been up for six hundred and twenty-two days). And so on. ... it seems that for certain machines, the entire backup process has restarted, numbering from 0, with more inaccessible backups. It seems to be turning into quite a mess and it seems the cleanup system is spiraling out of control and deleting all my files. Joy. I have attached a PNG of the current rrdtool graph of the 4-week Pool Size, which might give you an idea of what is happening. I have a feeling it will be rejected by the mailing list but I am CC:'ing you ... The private mail was rejected, we don't accept anything from AS14061, but I have it. Sometimes I think I never should have patched RRDTool to do those logarithimic graphs. :/It doesn't tell me much, but I don't like the look of those sudden steps in week 44. ... This is BackupPC 4.4.0. Again, this is the Docker container ... My preference for all backup machines is to avoid anything resembling containers, VMs, cloud, ... I want to see the lights flashing on some hard, bare metal - in the same rack that contains the UPS - and I want to be able to pick it up and carry it out to the Jeep if I have to. If I thought there's any risk that old data will get overwritten as a result of restarting the backup number count from 0 then I'd turn off the machine and install a new one. If the backups are valuable it's just not worth the risk, it would be a *lot* cheaper than my fee and it will Just Work. If you need to go to old backups later on you can get down and forensically dirty when you have your current backups in good shape. HTH -- 73, Ged. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Can't access older backups after upgrade
On 11/2/21 7:54 AM, G.W. Haywood via BackupPC-users wrote: Hi there, Hi G.W.! Thanks for responding. FWIW I wouldn't use NFS for anything that even vaguely bears a passing resemblance to a backup system. In my experience NFS is always flaky. Even if you do get it more or less under control, there always seem to be gotchas lurking in there. Namely? I use NFS for a lot of things. And apart from an abysmal security model (that can fortunately be hardened with a lot of extra work), I've had no complaints. One of the things to watch out for is indirection, where processes are using some configured value. This is particularly true when the value is a filesystem path. That might be helpful and it might not, in this case, I throw it in because at this stage I can't immediately think of anything else _concrete_ that might be relevant to your issue. You've done enough already, however, to give anybody, er, pause for thought. Indirection of configuration variables? I'm not so sure that's an issue with my config, but how would I know? My gut feeling is no, your backups will not be lost - unless you do something rash. But given that (1) you seem to think that your backups are important to you and (2) you've been messing about wantonly with the architecture and (3) it appears that this is your only backup then what you have done so far seems to me a bit on the rash side. How do you mean? Anything that MIGHT have involved writing or modification of data would have involved the running of: BackupPC_fixupBackupSummary, BackupPC_fsck, BackupPC_migrateV3toV4, BackupPC_nightly, or BackupPC_refCountUpdate, which in some cases I ran manually but in others, the system has been just left on overnight. There have been no rm's, mv's, or other form of data modification. So I would like to think that point (2) you make is inaccurate and would hope that in doing so my backups are indeed safe. However, as I've tried to look back at particular backup #s that used to have backups (or at least give me the error about missing files), it seems that for certain machines, the entire backup process has restarted, numbering from 0, with more inaccessible backups. It seems to be turning into quite a mess and it seems the cleanup system is spiraling out of control and deleting all my files. Joy. I have attached a PNG of the current rrdtool graph of the 4-week Pool Size, which might give you an idea of what is happening. I have a feeling it will be rejected by the mailing list but I am CC:'ing you directly. I am open to paying for one-on-one assistance with this. I would very much like to salvage this situation, if at all possible. Well I probably should't pass up an offer like that, but first things first: what version(s) of BackupPC are we talking about? My apologies, I usually provide this information. This is BackupPC 4.4.0. Again, this is the Docker container (adferrand/backuppc), which it turns out, requires a few Perl modules, which I discovered were not installed by default, once I attempted to run a few of the utility programs manually. They are now installed but I will need to fork and modify the Dockerfile in order for them to stick around. - acc ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
