You can use the DumpPreUserCmd and DumpPostUserCmd settings to manage lockfiles and
make sure backups from the aliased hosts cannot run at the same time.
You can separate them in the scheduler by manually starting them at different times,
or by disabling the automatic backups and using cron to start the backups
at
particular times.
On 4/8/2021 9:47 AM, Mike Hughes wrote:
Hi Dave,
You can always break a backup job into multiple backup 'hosts' by using
the ClientNameAlias setting. I create hosts based on the share or folder for each
job, then use the ClientNameAlias to point them to the same host.
-------------------------------------------------------------------------------------
*From:* Dave Sherohman <dave.sheroh...@ub.lu.se>
*Sent:* Thursday, April 8, 2021 8:22 AM
*To:* General list for user discussion, questions and support
<backuppc-users@lists.sourceforge.net>
*Subject:* [BackupPC-users] Handling machines too large to back themselves up
I have a server which I'm not able to back up because, apparently, it's
just too big.
If you remember me asking about synology's weird rsync a couple weeks ago, it's
that machine again. We finally solved the rsync issues by ditching the synology
rync entirely and installing one built from standard rsync source code and using
that instead. Using that, we were able to get one "full" backup, but
it missed a
bunch of files because we forgot to use sudo when we did it. (The synology rsync
is set up to run suid root and is hardcoded to not allow root to run it, so we had
to take sudo out for that, then forgot to add it back in when we switched to
standard rsync.)
Since then, every attempted backup has failed, either full or incremental, because
the synology is running out of memory:
This is the rsync child about to exec /usr/libexec/backuppc-rsync/rsync_bpc
Xfer PIDs are now 1228998,1229014
xferPids 1228998,1229014
ERROR: out of memory in receive_sums [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(118)
[sender=3.2.0dev]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0
sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 32863617 inode
rsync_bpc: [generator] write error: Broken pipe (32)
The poor little NAS has only 6G of RAM vs. 9.4 TB of files (configured as two
sharenames, /volume1 (8.5T) and /volume2 (885G) and doesn't seem up to the task of
updating that much at once via rsync.
Adding insult to injury, even a failed attempt to back it up causes the
bpc server
to take 45 minutes to copy the directory structure from the previous backup before
it even attempts to connect, and then 12-14 hours doing reference counts after it
finishes backing up nothing. Which makes trial-and-error painfully slow, since we
can only try one thing, at most, each day.
In our last attempt, I tried flipping the order of the RsyncShareNames to do
/volume2 first, thinking it might successfully back up the smaller share
successfully before running out of memory trying to process the larger one. It did
not run out of memory... but it did sit there for a full 24 hours with one CPU (out
of four) running pegged at 99% handling the rsync process before we finally put it
out of its misery. The bpc xferlog recorded that the connection was closed
unexpectedly (which is fair, since we killed the other end) after 3182 bytes were
received, so the client clearly hadn't started sending data yet. And
now, after
that attempt, the bpc server still lists the status as "refCnt #2" another 24 hours
after the client-side rsync was killed.
So, aside from adding RAM, is there anything else we can do to try to work around
this? Would it be possible to break this one backup down into smaller chunks that
are still recognized as a single host (so they run in sequence and don't get
scheduled concurrently), but don't require the client to diff large amounts of data
in one go, and maybe also speed up the reference counting a bit?
An "optimization" (or at least an option) to completely skip the reference count
updates after a backup fails with zero files received (and, therefore, no
new/changed references to worry about) might also not be a bad idea.
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
