Hi there,
On Mon, 28 Sep 2026, Patrick Begou wrote:
I'm using BackupPC 3.3.1 on old servers (10 years old) running in a
container in Proxmox VE. In a few month all of this will be replaced
with new hardware and more up to date versions.
Version 3.3.1 was released on 12 January 2015, more than eleven years
ago. Version 4 will be much more efficient in some areas. For major
differences between BackupPC V3 and V4 see
https://backuppc.github.io/backuppc/BackupPC.html#BackupPC-4.0
Version 3 doesn't use Perl XS for example, so a lot of the heavy-duty
rsync transfer stuff in V3 is done by interpreted Perl code. But in
BackupPC version 4 it's done in C (through Perl's XS) and is probably
an order of magnitude faster on like-for-like hardware.
This doesn't necessarily mean that you can't boost the performance of
V3 but it does mean that most people here - including me - will have
forgotten an awful lot about a BackupPC version which should really
have been pensioned off years ago.
Is the other software on the BackupPC server as old as BackupPC itself?
If so then you might have more to worry about than just BackupPC.
Latops (running linux) have larger and larger SSD and I reach a
situation where in the full working day, BackupPC does not succeed
anymore for a full backup of the client using rsync. Incremental
Backup works fine. From the BackupPC container I've just run a "rsync
-av ..." command and it is able to upload the whole user's files
(/home/xxx) in 67 minutes; But backupPC is unable to do this in 8 hours.
BackupPC does not limit a backup to something that can be completed on
any particular timescale, although there may be timeouts which need a
little thought. Perhaps you need to be creative in the configuration,
or just leave the computers switched on a while longer?
From the BackupPC container I've just run a "rsync -av ..."
command and it is able to upload the whole user's files (/home/xxx) in
67 minutes; But backupPC is unable to do this in 8 hours.
Correction: "But version 3 of BackupPC is unable to ...".
I do not know what BackupPc is doing exactly when it is running
"/usr/bin/perl /usr/share/BackupPC/bin/BackupPC_dump...."
BackupPC_dump takes what it needs from your configuration in order to
back up the host or hosts in the way that you specified in the config.
Part of that involves starting a 'transfer' program. The word 'Xfer'
is just an old-timer's abbreviation for 'transfer' and is used quite a
lot in BackupPC. That saves a little typing but in other ways I'm not
sure that it's particularly helpful. For any particular share, the
transfer can be one of tar, SMB or rsync. (Yes, that means that for a
single client you can use more than one transfer method.) The rest of
what BackupPC_dump does involves managing backup numbers, expiring old
backups and stuff like that which isn't very interesting.
For rsync transfers, to communicate with the client's rsync instance
BackupPC version 3 uses File::RsyncP. File::RsyncP is a Perl module.
Because it doesn't use XS, File::RsyncP does a lot of the processing
using the Perl interpreter. So it does it slowly, and there's really
no way around that other than upgrading to BackupPC version 4.
The rest of this paragraph applies to BackupPC v4. I can't remember
if it's relevant to version 3, so YMMV, but others may be interested.
The transfer log is in $Conf{TopDir}/pc/hostname/XferLOG.N.z where 'N'
is the backup number and .z signifies that the compression used in the
file is of the kind provided by Zlib; BackupPC_zcat can uncompress it.
You will find $conf{TopDir} in the main BackupPC configuration file.
To view logs I find that it's best to uncompress them to a new file in
e.g. $HOME using tools like grep and a pager on the uncompressed file.
When I look at lots of logs from many machines I tend to use a command
line something like
N=2451 H=alpha ; /usr/local/BackupPC/bin/BackupPC_zcat \
/var/lib/BackupPC/pc/$H/XferLOG.$N.z | less -S
which makes it relatively painless to choose one of many hosts and
backup numbers without too much cursor movement on the command line.
Be aware that in the interests of efficiency a lot of time is spent
compressing data from one RAM buffer to another, with no storage or
network IO activity during that time. Log file data is flushed to
mass storage when a buffer gets filled. You might almost be forgiven
for thinking that BackupPC is dead except that 'top' says it isn't.
The same sort of thing can happen with files that are already in the
BackupPC pool.
and if there is a way to improve the file transferts from the client
to backupPC before processing them (compression, deduplication...)
...
For rsync transfers *both* the client *and* the BackupPC server are
heavily involved in the process. It isn't all down to the BackupPC
server. For example if a client's file already exists on a BackupPC
server there are different ways of testing the file for what I'll call
'sameness'. One way is to simply check the file size and timestamp.
That's normally very fast. Another way is to read the whole file, at
both ends of the connection, calculating checksums on both client and
server as the read progresses, and then comparing the checksums. This
is time consuming, especially if it's done by the Perl interpreter on
the BackupPC server as it will be in BackupPC version 3. You can use
the rsync configuration in BackupPC, or at least in V3 I *think* you
can, to decide the type of sameness test is used. Beware of possible
downsides to that, see the docs.
... at this time some clients have no valid backups for months.
You said that incremental backups work fine. So do you not have some
sort of backup? Some of my machines only do a full backup monthly.
Do you stagger the fulls/incrementals for different clients so that
the clients do not all need to start a full backup on the same day?
I look at the typology of the users files, there are folders with
numerous subfolders and small files (ex: .julia folder) and other
folders with few large files (ex: simulations results). But
generally it is not possible to have trees of folders with one or
the other typology only.
As far as BackupPC is concerned I don't think that there is much to be
gained from this train of thought, but there may well be some mileage
in looking at the partitioning of your shares. Again if you show us
your configuration we may be able to help with that. I like to use
rsync 'modules' on the clients to break backups into manageable parts.
Data volume is around 200GB.
Your network seems reasonably fast, that doesn't sound like a big ask
for a BackupPC server - at least one that's less than about ten years
old. The network here (like yours, mostly 1Gbit/s) has amongst its
backup clients several machines each of which individually needs to
back up more than twice that amount of data. The total data is around
20 Terabytes before pooling and compression. Of course it's BackupPC
V4 that does the backups. They aren't fast boxes, many of them only
1.5GHz 8GByte Pi4B. The main BackupPC server is a Pi4B. There are a
couple of others on faster machines, but they're mostly for testing.
To conclude I suspect that you could improve performance[*] with some
judicious configuration. But you really should be puting that effort
into upgrading first if for no other reason than that, in the 2020s,
nobody will put much effort into supporting 3.3.1 installations.
[*] Not necessarily pure speed - think about what gets done when.
