Re: [BackupPC-users] Full backups taking a very long time
On Sat, Mar 13, 2021 at 9:25 AM Sorin Srbu wrote: > > > Maybe some OS defaults changed at the OS-upgrade? > Or, if we are getting down to wild guesses, check that the network interfaces are running at gigabit, full duplex all the way through. A driver update might make the system more sensitive to out-of-spec cables or something and auto-switch down. -- Les Mikesell lesmikes...@gmail.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
On Fri, 2021-03-12 at 12:59 +, David Williams wrote: > No, I don’t have a lot of small files. As I mentioned, the full backups > weren’t taking that long before the upgrade to Ubuntu 20.04. Not sure why an > OS upgrade would severely increase the time to back up. The machine, hard > drives (SSD’s) are all the same. > > The version of BackupPC that comes with Ubuntu 20.04 is 3.3.2-3 > > Regards, > _ > Dave Williams > > > On Mar 11, 2021, 10:27 AM -0500, backu...@kosowsky.org > , wrote: > > Sorin Srbu wrote at about 08:31:35 +0100 on Thursday, March 11, 2021: > > > On Wed, 2021-03-10 at 14:04 +, David Williams wrote: > > > > I have recently upgraded to Ubuntu 20.04 and since then I have noticed > > > > that my full backups are taking much longer than they used to do. I’m > > > > only using backuppc to bakup two machines at home. The Ubuntu machine > > > > and a Mac laptop. I don’t recall exactly how long the full backups were > > > > taking previously, but now they are taking close to 21 hours. The > > > > content on both machines hasn’t changed much at all since the upgrade > > > > so I was surprised by the increase in time. > > > > > > > > A full backup on the Linux machine is around 892MB. This is the local > > > > machine that Backuppc is installed on. The drive that the backups are > > > > stored on is an SSD as are most, if not all (sorry can’t remember) of > > > > the drives in the Linux box. Backup method is tar. > > > > > > > > A full backup on the Mac laptop is around 700MB. It’s connected to the > > > > same router as the Linux machine via ethernet. Backup method is rsync. > > > > > > > > I’m not sure how to troubleshoot this increase in timing so any help > > > > would be much appreciated. > > > > > > I saw this happening when small files are backed up. > > > > Unless there is a truly pathological number of small files (think tens > > if not hundreds of millions), I don't think you can explain a 21 hour > > backup period. > > > > My Raspberry PI 4 on a home network backs up my Ubuntu 18.04 with > > 2.7GB and 321K files in under 12 minutes. And I think that is with > > several simultaneous backups. Maybe some OS defaults changed at the OS-upgrade? -- Kind regards, Sorin Srbu Find my OpenPGP public key here: https://cloud.srbu.se/index.php/s/KeEsCCDsG7PZG7N signature.asc Description: This is a digitally signed message part ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
No, I don’t have a lot of small files. As I mentioned, the full backups weren’t taking that long before the upgrade to Ubuntu 20.04. Not sure why an OS upgrade would severely increase the time to back up. The machine, hard drives (SSD’s) are all the same. The version of BackupPC that comes with Ubuntu 20.04 is 3.3.2-3 Regards, _ Dave Williams [cid:CD2541C197494118AC1AF4341C4B718C] On Mar 11, 2021, 10:27 AM -0500, backu...@kosowsky.org , wrote: Sorin Srbu wrote at about 08:31:35 +0100 on Thursday, March 11, 2021: On Wed, 2021-03-10 at 14:04 +, David Williams wrote: I have recently upgraded to Ubuntu 20.04 and since then I have noticed that my full backups are taking much longer than they used to do. I’m only using backuppc to bakup two machines at home. The Ubuntu machine and a Mac laptop. I don’t recall exactly how long the full backups were taking previously, but now they are taking close to 21 hours. The content on both machines hasn’t changed much at all since the upgrade so I was surprised by the increase in time. A full backup on the Linux machine is around 892MB. This is the local machine that Backuppc is installed on. The drive that the backups are stored on is an SSD as are most, if not all (sorry can’t remember) of the drives in the Linux box. Backup method is tar. A full backup on the Mac laptop is around 700MB. It’s connected to the same router as the Linux machine via ethernet. Backup method is rsync. I’m not sure how to troubleshoot this increase in timing so any help would be much appreciated. I saw this happening when small files are backed up. Unless there is a truly pathological number of small files (think tens if not hundreds of millions), I don't think you can explain a 21 hour backup period. My Raspberry PI 4 on a home network backs up my Ubuntu 18.04 with 2.7GB and 321K files in under 12 minutes. And I think that is with several simultaneous backups. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
On Thu, 2021-03-11 at 10:26 -0500, backu...@kosowsky.org wrote: > Sorin Srbu wrote at about 08:31:35 +0100 on Thursday, March 11, 2021: > > On Wed, 2021-03-10 at 14:04 +, David Williams wrote: > > > I have recently upgraded to Ubuntu 20.04 and since then I have noticed > that my full backups are taking much longer than they used to do. I’m only > using backuppc to bakup two machines at home. The Ubuntu machine and a Mac > laptop. I don’t recall exactly how long the full backups were taking > previously, but now they are taking close to 21 hours. The content on both > machines hasn’t changed much at all since the upgrade so I was surprised by > the increase in time. > > > > > > A full backup on the Linux machine is around 892MB. This is the local > machine that Backuppc is installed on. The drive that the backups are stored > on is an SSD as are most, if not all (sorry can’t remember) of the drives in > the Linux box. Backup method is tar. > > > > > > A full backup on the Mac laptop is around 700MB. It’s connected to the > same router as the Linux machine via ethernet. Backup method is rsync. > > > > > > I’m not sure how to troubleshoot this increase in timing so any help > would be much appreciated. > > > > I saw this happening when small files are backed up. > > Unless there is a truly pathological number of small files (think tens > if not hundreds of millions), I don't think you can explain a 21 hour > backup period. > > My Raspberry PI 4 on a home network backs up my Ubuntu 18.04 with > 2.7GB and 321K files in under 12 minutes. And I think that is with > several simultaneous backups. I had that particular case going on at my previous work. Twentyish linux workstations and all having homefolders with tens of millions of small files. I resorted to decreasing the amount of concurrently backed up computers for a small increase in overall speed as well as excluding some folders. Some clients took hours to complete. Not 21 hours, but maybe a quarter of that. We later moved to faster networks and harddrives which helped too. -- Kind regards, Sorin Srbu Find my OpenPGP public key here: https://cloud.srbu.se/index.php/s/KeEsCCDsG7PZG7N signature.asc Description: This is a digitally signed message part ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
Hi Dave, does each client individually take so long, or is it just one? (Or perhaps one client is taking almost all resources, so the other one is slow.) If both clients suddenly have problems, an issue with the BackupPC server is likely. Otherwise, I've seen similar issues when suddenly some service on the client decided to include a virtual memory map file somewhere in /var/run. Which pretends to be of **huge** size, like 2 TB or something. And I didn't use the -x, --one-file-system flag for rsync. The file mostly consists of (implicitly represented) zeros and compresses really nicely, so BackupPC doesn't choke on it, technically; but the processing takes forever, and eventually it always hit some timeout. So I recommend a `sudo find /share -size +1G` to find potential huge files that you don't expect to be in the backups. Or try `sudo ls -l /proc/$(pgrep rsync)/fd` on the clients during backup and try to see whether rsync still progresses, and if not, on which file it idles. Same for tar, obviously. Cheers, Alex On 3/10/21 3:04 PM, David Williams wrote: > I have recently upgraded to Ubuntu 20.04 and since then I have noticed that > my full backups are taking much longer than they used to do. I’m only using > backuppc to bakup two machines at home. The Ubuntu machine and a Mac laptop. > I don’t recall exactly how long the full backups were taking previously, but > now they are taking close to 21 hours. The content on both machines hasn’t > changed much at all since the upgrade so I was surprised by the increase in > time. > > A full backup on the Linux machine is around 892MB. This is the local > machine that Backuppc is installed on. The drive that the backups are stored > on is an SSD as are most, if not all (sorry can’t remember) of the drives in > the Linux box. Backup method is tar. > > A full backup on the Mac laptop is around 700MB. It’s connected to the same > router as the Linux machine via ethernet. Backup method is rsync. > > I’m not sure how to troubleshoot this increase in timing so any help would be > much appreciated. > > Regards, > _ > *Dave Williams* > > > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ > smime.p7s Description: S/MIME Cryptographic Signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
Sorin Srbu wrote at about 08:31:35 +0100 on Thursday, March 11, 2021: > On Wed, 2021-03-10 at 14:04 +, David Williams wrote: > > I have recently upgraded to Ubuntu 20.04 and since then I have noticed > > that my full backups are taking much longer than they used to do. I’m > > only using backuppc to bakup two machines at home. The Ubuntu machine and > > a Mac laptop. I don’t recall exactly how long the full backups were > > taking previously, but now they are taking close to 21 hours. The content > > on both machines hasn’t changed much at all since the upgrade so I was > > surprised by the increase in time. > > > > A full backup on the Linux machine is around 892MB. This is the local > > machine that Backuppc is installed on. The drive that the backups are > > stored on is an SSD as are most, if not all (sorry can’t remember) of the > > drives in the Linux box. Backup method is tar. > > > > A full backup on the Mac laptop is around 700MB. It’s connected to the > > same router as the Linux machine via ethernet. Backup method is rsync. > > > > I’m not sure how to troubleshoot this increase in timing so any help would > > be much appreciated. > > I saw this happening when small files are backed up. Unless there is a truly pathological number of small files (think tens if not hundreds of millions), I don't think you can explain a 21 hour backup period. My Raspberry PI 4 on a home network backs up my Ubuntu 18.04 with 2.7GB and 321K files in under 12 minutes. And I think that is with several simultaneous backups. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Full backups taking a very long time
On Wed, 2021-03-10 at 14:04 +, David Williams wrote: > I have recently upgraded to Ubuntu 20.04 and since then I have noticed that > my full backups are taking much longer than they used to do. I’m only using > backuppc to bakup two machines at home. The Ubuntu machine and a Mac laptop. > I don’t recall exactly how long the full backups were taking previously, but > now they are taking close to 21 hours. The content on both machines hasn’t > changed much at all since the upgrade so I was surprised by the increase in > time. > > A full backup on the Linux machine is around 892MB. This is the local > machine that Backuppc is installed on. The drive that the backups are stored > on is an SSD as are most, if not all (sorry can’t remember) of the drives in > the Linux box. Backup method is tar. > > A full backup on the Mac laptop is around 700MB. It’s connected to the same > router as the Linux machine via ethernet. Backup method is rsync. > > I’m not sure how to troubleshoot this increase in timing so any help would be > much appreciated. I saw this happening when small files are backed up. -- Kind regards, Sorin Srbu Find my OpenPGP public key here: https://cloud.srbu.se/index.php/s/KeEsCCDsG7PZG7N signature.asc Description: This is a digitally signed message part ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Full backups taking a very long time
I have recently upgraded to Ubuntu 20.04 and since then I have noticed that my full backups are taking much longer than they used to do. I’m only using backuppc to bakup two machines at home. The Ubuntu machine and a Mac laptop. I don’t recall exactly how long the full backups were taking previously, but now they are taking close to 21 hours. The content on both machines hasn’t changed much at all since the upgrade so I was surprised by the increase in time. A full backup on the Linux machine is around 892MB. This is the local machine that Backuppc is installed on. The drive that the backups are stored on is an SSD as are most, if not all (sorry can’t remember) of the drives in the Linux box. Backup method is tar. A full backup on the Mac laptop is around 700MB. It’s connected to the same router as the Linux machine via ethernet. Backup method is rsync. I’m not sure how to troubleshoot this increase in timing so any help would be much appreciated. Regards, _ Dave Williams [cid:A6010CA784C04BC0A13ABC6690456690] ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
