Package: backup-manager Version: 0.7.10.1-2 Severity: grave Justification: renders package unusable
Dear Maintainer,
I had suspected for a while there was an issue with the way backup-manager
handles incremental backups. The issue is related with the way older archives
are deleted when BM_ARCHIVE_TTL is reached.
I now have built a (albeit crude) test suite, attached, to test that
behaviour, and as I feared, backup-manager causes data loss. I strongly urge
anyone using backup-manager to avoid using tarball-incremental mode until
this bug is sorted out.
backup-manager purges backup files that are older than "today minus
BM_ARCHIVE_TTL days." For incremental tarballs, it will, however, never
delete the latest "master" backup until a new master will have been created -
but it will delete incrementals in between the said master and
TODAY minus BM_ARCHIVE_TTL.
Note that new masters will be only created if we are precisely on
BM_TARBALLINC_MASTERDATEVALUE [day of the week or day of the month].
If the backup happens another day, it will do an incremental backup, based
on the "backup-name.incremental.bin" data file, regardless of whether the
current master backup is older than BM_ARCHIVE_TTL or not. However,
incremental.bin is not
updated following the deletion of previous incremental tarballs "in between"
the old master and the new incremental. Therefore the new incremental
backups do not store everything required to maintain backup consistency.
So one can have a very old master, some recent incremental backups, and
a very big 'hole' in between - and all files created in that 'hole' will
be lost if one tries to restore from the incremental (also all files modified
in that period will be reverted to the version stored in the old master).
The situation is very likely to happen:
- if you automate backups every day, if and only if
BM_ARCHIVE_TTL < 7 (for weekly masters) or BM_ARCHIVE_TTL < 31 (for monthly
masters), since that would be the situation where incrementals would be
deleted before a new master is created,
- if you automate backups weekly with a BM_TARBALLINC_MASTERDATETYPE set to
"monthly" as BM_TARBALLINC_MASTERDATEVALUE may be missed,
- if you do manual backups and forget to start it regularly on the day you
have set the master archive creation.
I enclose to this bug report a test suite containing:
- in bindir/ : a set of scripts that override /bin/date in order to
simulate several days of a given week
- in bindir/ : two one-line modifications of /usr/sbin/backup-manager [line 36]
and /usr/bin/backup-manager-purge [line 244]
which were unfortunately necessary to call the special "date"
instead of the standard perl time()
- two empty test directories testdir/ and backupdir/
- a custom backup-manager.conf used by the test script, with BM_ARCHIVE_TTL
set to 2, which should be modified by the tester (some paths to change)
- a test.sh script that should be modified (TESTDIR, BACKUPNAME to be changed)
in order to run it
test.sh simulates a week going from monday to friday, with a master backup
set to monday. It generates or modifies files in testdir/ prior to each
backup to simulate file system changes, and those files are stored consecutively
as master and incremental backups into backupdir/
On monday:
- it creates three files, "never_modified.txt" with contents "never modified";
"modified_every_day.txt", "modified_until_tuesday.txt" with "monday" as their
content.
- calls backup-manager
On tuesday:
- it creates "created_tuesday.txt" with content "created tuesday", and writes
"tuesday" as the content of "modified_until_tuesday.txt" and
"modified_every_day.txt"
- calls backup-manager
On wednesday:
- writes "wednesday" as the content of "modified_every_day.txt"
- calls bm
On thursday:
- calls bm
On friday:
- idem
The test script then unpack all the backups that should be in the directory
according to the behaviour of backup-manager explained above, considering
BM_ARCHIVE_TTL is set to 2 for testing purposes, in that order: the master
backup of monday, the incremental backup of thursday, the incremental backup
of friday.
it then compares the resulting files with the files in testdir/
What is expected:
the restored backup should contain:
never_modified.txt with content "never modified"
modified_until_tuesday.txt with content "tuesday"
created_tuesday.txt with content "created tuesday"
modified_every_day.txt with content "friday"
What actually happens:
the restored backup contains:
never_modified.txt with content "never modified"
modified_until_tuesday.txt with content "monday"
modified_every_day.txt with content "friday"
There are several ways to resolve this issue:
- either do not delete any incremental backups as long as a new master has
not been created, even if they are over BM_ARCHIVE_TTL
- figure out a way to update backupname.incremental.bin when deleting
the old "in-between" incrementals so that its content reflects that deletion
- force the generation of a new master as soon as the current one is older than
BM_ARCHIVE_TTL, even if that is not the "right time of the week/month" to
do so.
Many thanks in advance and good luck :)
mathieu
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation?
* What exactly did you do (or not do) that was effective (or
ineffective)?
* What was the outcome of this action?
* What outcome did you expect instead?
*** End of the template - remove these template lines ***
-- System Information:
Debian Release: 8.0
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=fr_FR.utf8, LC_CTYPE=fr_FR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages backup-manager depends on:
ii debconf [debconf-2.0] 1.5.56
ii perl 5.20.2-3
ii ucf 3.0030
backup-manager recommends no packages.
Versions of packages backup-manager suggests:
ii anacron 2.3-23
pn backup-manager-doc <none>
pn dar <none>
pn dvd+rw-tools <none>
ii genisoimage 9:1.1.11-3
ii gettext-base 0.19.3-2
pn libnet-amazon-s3-perl <none>
ii openssh-client 1:6.7p1-5
ii wodim 9:1.1.11-3
ii zip 3.0-8
-- debconf information excluded
test-backup-manager.tgz
Description: application/gzip
