Package: duplicity Version: 0.5.16-1 Severity: normal (You'll want to forward this upstream probably.)
duplicity was triggering the OOM killer when ran on a machine with 128
mb of ram. This machine has a 64 gb flash filesystem, though only
a few dozen MB of that are being backed up, and has no swap.
The problem is that while duplicity itself only uses about 5% of the
memory, it builds up an extrordinary number of gpg processes. Each
consumes about 1 mb of unshared memory (on armel). On a recent run, I
saw duplicity have 40+ gpg processes running at once. Cumulatively,
these processes eventually eat so much memory that the OOM killer runs.
duplicity is being ran as follows:
duplicity --allow-source-mismatch --archive-dir /var/backups/duplicity
--no-print-statistics --scp-command scp --sftp-command sftp --encrypt-key
F7915DF7 --sign-key F7915DF7 scp://j...@turtle//home/joey/lib/backup/box
The gpg processes look like this in top:
28858 root 39 19 3488 1484 1164 S 0.0 1.2 0:00.54 gpg
28864 root 39 19 3488 1484 1164 S 0.0 1.2 0:00.54 gpg
28867 root 39 19 3488 1484 1164 S 0.0 1.2 0:00.57 gpg
28870 root 39 19 3488 1484 1164 S 0.0 1.2 0:00.55 gpg
28873 root 39 19 3488 1484 1164 S 0.0 1.2 0:00.55 gpg
28876 root 39 19 3488 1484 1164 S 18.8 1.2 0:00.58 gpg
28855 root 39 19 3676 1480 1148 S 0.0 1.2 0:00.57 gpg
...
And like this in ps fax:
28864 pts/2 SNL+ 0:00 | | \_ gpg
--status-fd 13 --passphrase-fd 17 --logger-fd 11 --batch --no-tty --default-key
F7915DF7 --no-secmem-warning --decrypt
28867 pts/2 SNL+ 0:00 | | \_ gpg
--status-fd 16 --passphrase-fd 20 --logger-fd 14 --batch --no-tty --default-key
F7915DF7 --no-secmem-warning --decrypt
28870 pts/2 SNL+ 0:00 | | \_ gpg
--status-fd 19 --passphrase-fd 23 --logger-fd 17 --batch --no-tty --default-key
F7915DF7 --no-secmem-warning --decrypt
...
Note that these are decompression processes, while duplicity is being
used to create a new incremental backup. I assume this is because it
needs to read compressed metadata from older backups first. (While
actually doing the backup, duplicity seems to only keep two gpg
compression processes running at a time.)
A nasty side effect of this is that duplicity was installed on this
machine months ago, and worked fine for a long time, until enough data
accumulated that it started to OOM and kill important services on the
machine. Thus we have a backup service that acts as a slow time bomb.
Now I have to consider how long my other, larger machines might last
until duplicity starts to eat them alive. :-/
Internally, duplicity in gpg.py creates GPGFile objects that represent a
file that gets decrypted on the fly using gpg. So in the init method of
each object, gpg is started, and it keeps running until the file is
closed, even once the file stops being accessed.
I suspect this is the code at fault for opening so many files concurrently:
def get_fileobjs(self, time = None):
"""Return ordered list of signature fileobjs opened for reading,
optionally at a certain time"""
assert self.fullsig
if self.archive_dir: # local
def filename_to_fileobj(filename):
"""Open filename in archive_dir, return filtered fileobj"""
sig_dp = path.DupPath(self.archive_dir.name, (filename,))
return sig_dp.filtered_open("rb")
else:
filename_to_fileobj = self.backend.get_fileobj_read
return map(filename_to_fileobj, self.get_filenames(time))
Unfortunatly, the code seems to be written with the assumption that
passing around enormous lists of open files is a good idea[1]. This
seems to make it hard to fix -- the choices seem to be either to rewrite
lots of code that uses these FD lists to instead use filename lists; or
to somehow change the GPGFile objects so that they do not keep gpg
running for the entire lifetime of the FD.
--
see shy jo
[1] Of course, this is problimatic even without them being associated
with gpg, for example I found this bug report about duplicity
running over a 1024 FD limit: https://savannah.nongnu.org/bugs/?24825
It was "fixed" by making it pre-check the limit.
signature.asc
Description: Digital signature
