This is a known issue with duplicity. You will need to increase your
systems number of open files since it's set so low. I'd suggest 16384
or higher if you have a lot of incrementals. I'd also suggest limiting
the number of incrementals and doing more full backups.
** Also affects: duplicity (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Desktop
Packages, which is subscribed to duplicity in Ubuntu.
https://bugs.launchpad.net/bugs/1878560
Title:
Restore fails with “too many open files”
Status in Duplicity:
New
Status in duplicity package in Ubuntu:
New
Bug description:
I use duplicity with Backblaze B2, have an incremental backup every
hour and a full backup every 90 days. I recently tried to restore a
file to the state that it had 14 days ago. Eventually it failed saying
“too many open files”.
Apparently the last full backup was 2020-03-08 15:00:05Z. So that
means that _a lot_ of incremental backups needed to be resolved. These
are the last lines of output:
Get: duplicity-inc.20200414T190005Z.to.20200414T200005Z.vol1.difftar.gpg ->
temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252
Verarbeitung des lokalen Manifest
b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T200005Z.to.20200414T210005Z.manifest'
(7297)
1 Volumen in Manifest gefunden
Get: duplicity-inc.20200414T200005Z.to.20200414T210005Z.vol1.difftar.gpg ->
temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253
Verarbeitung des lokalen Manifest
b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T210005Z.to.20200414T220005Z.manifest'
(5983)
1 Volumen in Manifest gefunden
Get: duplicity-inc.20200414T210005Z.to.20200414T220005Z.vol1.difftar.gpg ->
temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254
Traceback (innermost last):
File "/usr/bin/duplicity", line 100, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 86, in with_tempdir
fn()
File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1538,
in main
do_backup(action)
File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1618,
in do_backup
restore(col_stats)
File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 723,
in restore
if not patchdir.Write_ROPaths(globals.local_path,
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 578,
in Write_ROPaths
for ropath in rop_iter:
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 541,
in integrate_patch_iters
for patch_seq in collated:
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 407,
in yield_tuples
setrorps(overflow, elems)
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 396,
in setrorps
elems[i] = next(iter_list[i])
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 126,
in difftar2path_iter
tarinfo_list = [next(tar_iter)]
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 355,
in __next__
self.set_tarfile()
File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 348,
in set_tarfile
self.current_fp = next(self.fileobj_iter)
File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 759,
in get_fileobj_iter
yield restore_get_enc_fileobj(backup_set.backend,
File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 804,
in restore_get_enc_fileobj
fileobj = tdp.filtered_open_with_delete(u"rb")
File "/usr/lib64/python3.8/site-packages/duplicity/dup_temp.py", line 125,
in filtered_open_with_delete
fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
File "/usr/lib64/python3.8/site-packages/duplicity/path.py", line 802, in
filtered_open
return gpg.GPGFile(False, self, gpg_profile)
File "/usr/lib64/python3.8/site-packages/duplicity/gpg.py", line 225, in
__init__
p1 = gnupg.run([u'--decrypt'], create_fhs=gnupg_fhs,
File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line
374, in run
process = self._attach_fork_exec(gnupg_commands, args,
File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line
403, in _attach_fork_exec
pipe = os.pipe()
OSError: [Errno 24] Too many open files
I would assume that it stumbles over the 1024 file descriptor limit and it
just cannot resolve a backup with that many incremental backups. Basically with
the current state of the software my incremental backups are worthless because
I cannot restore. Is there something that I can do except doing less frequent
incremental updates and more full backups?
To manage notifications about this bug go to:
https://bugs.launchpad.net/duplicity/+bug/1878560/+subscriptions
--
Mailing list: https://launchpad.net/~desktop-packages
Post to : desktop-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~desktop-packages
More help : https://help.launchpad.net/ListHelp
