Hi, Dr. Boris Neubert wrote on 2014-09-29 20:16:56 +0200 [[BackupPC-users] Backup over slow link: endless loop or just missing patience?]: > [...] > The uplink is a slow DSL line with a maximum of 640 kBit/s towards the > BackupPC server. BackupPC server is connected to NAS via openVPN. > Transfer method is rsyncd. > [...] > Next data set added was my home directory with 15 GB.
at the maximum theoretical throughput that would make 54.6 hours. I wouldn't expect more than half the throughput in real world conditions, so that would be more like 110 hours. More even if the incrementals of your other backups are competing for bandwidth. If you say your 1 GB backup took "a few days", I'd expect 15 GB to take 15 times "a few days" under similar circumstances. I would advise against raising ClientTimeout to make it try to complete in one run. > [...] > What bothers me are the fatal errors (signal= ALRM) in combination with > the last lines of the "Last bad XferLOG" [...] > > Not saving this as a partial backup since it has fewer files than the prior > one (got 27503 and 27503 files versus 27628) I would normally *expect* BackupPC to use the previous partial as a reference, meaning it should check but not transfer the saved files and then continue with the rest. That should give you gradual progress, until finally the 15 GB complete. But that does not seem to be happening for you. An obvious cause would be if there were one single large file that does not complete within the timeout window, though then I'd expect the numbers to match ("27503 files versus 27503"). Another would be a large amount of changes in the first part - having saved the data yesterday does not help if it needs to be retransferred today because it has changed. A combination of both might be what you are seeing (similar but not equal numbers). > Does this mean that I am actually stuck in an endless loop because there > is a certain point in time or amount of data beyond which the backup > does not go? Apparently. It should be noted that partial backups are - as far as I understand it - meant to speed up a second attempt at a full rsync(d) backup if the first failed partway through (power failure, link failure, ...), *not* for piecing together "huge" data sets. Note that there is a setting $Conf{PartialAgeMax} which will limit the use of a partial. The first thing to do is to make sure that your plan is feasible at all. If the amount of daily changes exceeds the bandwidth available for backup, then that's not the case. "A few days" for 1 GB doesn't sound promising! You have probably done the maths and concluded that it should work. I would try any of the following: 1.) Limit the size of the backup to something that will complete within the alarm timeout. I'd start with a *small* number - perhaps only 100 MB at first (or even less), to get a feeling for how long things really take. You do that with (additional) excludes. Once you have that backup, add more to your data set by gradually removing excludes. Do it in small enough steps that your backups will succeed (without generating partials!). Run only full backups (or alternating full and incremental). Two consequential incrementals would let the second one re-transfer the data the first one has covered (unless you are using IncrLevels, but let's not complicate things). If your backup completes after an hour, you can always start the next one manually rather than waiting for the scheduler. That's probably a slow and cumbersome process, but you seem to be stretching the limits of what is possible with your internet connection. You *have* set up meaningful excludes, I suppose? There's a lot of temporary data in a home directory that doesn't warrant backing up (.thumbnails, browser cache directories, mailer IMAP caches, ...) and, to make things worse, that tends to change frequently. Finding and excluding all of this stuff is a somewhat surgical operation, but with your bandwidth constraints I can't see a way around it. A [somewhat outdated] excerpt from my server excludes: '/*/.thumbnails', '/*/.evolution/mail/imap/user@host*', '/*/.evolution/cache', '/*/evolution/mail/imap/user@host*', '/*/.galeon/mozilla/galeon/Cache', '/*/.mozilla/*/*/Cache', '/*/.gnome2/epiphany/mozilla/epiphany/Cache', '/*/.adobe/Acrobat/*/Cache', '/*/.nautilus/thumbnails', '/*/.Trash', '/*/.googleearth/Cache' (my share is /home, so you'd probably need to remove the leading '/*'). You get the picture? There are a lot of caches around, partly hidden in unexpected places. Use a program that graphically displays your disk usage (something like 'filelight' perhaps) to find them. 2.) Create a copy of your data set on the VM with rsync manually. That gives you far more control over what is happening, and you might be able to use compression (-z option). Then import that backup with the ClientAlias trick occasionally discussed here (you'd temporarily need rsyncd on the BackupPC server). 3.) Find a neighbour with a faster internet connection who is willing to let you use it for some time (presuming you can establish a WLAN or cable connection to his network). VDSL can easily have 10 MBit/s uplink. A speed increase by a factor of 16 would really help you :-). You might even think of upgrading your own DSL connection, though that could take time and is not possible in all circumstances. 4.) Take your NAS (or a copy of your data and your OpenVPN key) to a place with more bandwidth for the initial backup. That's about all I can think of right now. Hope some of that helps. Regards, Holger ------------------------------------------------------------------------------ Slashdot TV. Videos for Nerds. Stuff that Matters. http://pubads.g.doubleclick.net/gampad/clk?id=160591471&iu=/4140/ostg.clktrk _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/