Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Jonathan Hutchins]

Does it occur to you that the reason for having a "testing" release is
precisely so that problems like this can be found and fixed, and that this
is why it's not smart to run testing on essential production machines?

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Lupe Christoph]

On Saturday, 2017-01-28 at 14:51:19 +, Holger Levsen wrote:
> On Sat, Jan 28, 2017 at 03:04:56PM +0100, Daniel Reichelt wrote:
> > I highly suspect this stems from packages' rules files supporting
> > reproducible builds.

> I rather think this is due to binNMUs not modifying debian/changelog…
> (in the source package while it's modified in the binary packages…)

This is completely counter intuitive. I'm using rsnapshot to backup a
few private machines. I will have to set up separate rsnapshots for
those parts of the backup that suffer from this , to avoid
bogging down the data parts by unnecessary checksum calculations.

This problem may affect many other backups too. Did anybody research
backup programs before this  was introduced to Debian?

The servers are all Debian, except for my home router which is running
OpenWRT. But for curiosity, does anybody know if Ubuntu is using the
same , or are they doing the usual, i.e. not follow Debian?

Thanks,
Lupe Christoph
-- 
| As everyone knows, it was predicted that the world would end last   |
| Wednesday at 10:00 PST.  Since there appears to be a world in existence |
| now, the entire universe must therefore have been recreated, complete   |
| with an apparent "history", last *Thursday*.  QED.  |
| Seanna Watson, <1992nov2.165142.11...@bcrka451.bnr.ca>  |

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Bob Weber]

As a user I have run into this with X11 files myself.  I use rsnapshot to backup
the root partition to a location on /home mounted on its own much larger
partition before I do upgrades.  A while back when xorg was crashing a lot I had
to restore from this backup.  I routinely use "debsums -ca" after an upgrade to
check OS files against their hashes and I discovered this strange occurrence
with files having the same date/time and size but different hashes.  Luckily I
have a server running apt-cacher-ng so I had the older deb files to re-install
to fix the problem.

So as a user this a BIG pain in the rear.  I also tried the rsync checksum
option but as you noted it is much more time consuming.  Are these files
actually equivalent but with their code segments scrambled for security?  I was
actually running xorg with the "bad" files for a while before I found them with
debsums.  From a user point of view this is totally inexcusable.  I hope that a
solution can be found to fix this ... even if it is just incrementing the second
by one.



*...Bob*
On 01/28/2017 08:11 AM, Adam Warner wrote:
> Hi all,
>
> rsync typically detects file modifications by comparing exact
> modification time and size of each identically named file in the source
> and destination locations.
>
> An rsync backup can be surreptitiously corrupted by modifying a source
> file's content while keeping its size and modification time the same.
>
> If some program is doing this then one way for rsync to detect the
> modification is by appending the --checksum option. This requires every
> file in the source and destination to be fully read. This can be many
> orders of magnitude slower than a "quick check".
>
> If you have been using rsync without --checksum to back up your Debian
> partitions then your backups are likely corrupted.
>
> Some packages are being generated with files containing exactly the
> same size and timestamps even though the contents of those files are
> different. This is how the data corruption arises:
>
> 1. You use rsync to back up your OS partition.
> 2. You perform package upgrades. Some files are replaced with different
> content but have the same size and modification time.
> 3. You use rsync to back up your OS partition to the same destination.
> The modified files with the same size and modification time are
> skipped. Your backup is now corrupt (you have old files mixed with new
> files).
>
> Here's a recent example of a package upgrade causing this:
>
> 
> 
>
> If you're tracking unstable you may have recently upgraded from
> libqt5concurrent5_5.7.1+dfsg-3_amd64.deb to
> libqt5concurrent5_5.7.1+dfsg-3+b1_amd64.deb (a subsequent binary non-
> maintainer upload).
>
> Here are the contents of both packages:
> $ dpkg -c libqt5concurrent5_5.7.1+dfsg-3_amd64.deb 
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/x86_64-linux-gnu/
> -rw-r--r-- root/root 27352 2017-01-12 04:14 
> ./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7.1
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/doc/
> drwxr-xr-x root/root 0 2017-01-12 04:14 
> ./usr/share/doc/libqt5concurrent5/
> -rw-r--r-- root/root  1196 2016-12-01 21:17 
> ./usr/share/doc/libqt5concurrent5/LGPL_EXCEPTION.txt
> -rw-r--r-- root/root 18232 2017-01-12 04:14 
> ./usr/share/doc/libqt5concurrent5/changelog.Debian.gz
> -rw-r--r-- root/root  3792 2016-12-01 21:17 
> ./usr/share/doc/libqt5concurrent5/changelog.gz
> -rw-r--r-- root/root103466 2017-01-12 04:14 
> ./usr/share/doc/libqt5concurrent5/copyright
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/overrides/
> -rw-r--r-- root/root   230 2017-01-12 04:14 
> ./usr/share/lintian/overrides/libqt5concurrent5
> lrwxrwxrwx root/root 0 2017-01-12 04:14 
> ./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5 -> libQt5Concurrent.so.5.7.1
> lrwxrwxrwx root/root 0 2017-01-12 04:14 
> ./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7 -> 
> libQt5Concurrent.so.5.7.1
>
> $ dpkg -c libqt5concurrent5_5.7.1+dfsg-3+b1_amd64.deb 
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/x86_64-linux-gnu/
> -rw-r--r-- root/root 27352 2017-01-12 04:14 
> ./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7.1
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/
> drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/doc/
> drwxr-xr-x 

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Daniel Reichelt]

On 01/28/2017 03:51 PM, Holger Levsen wrote:
> On Sat, Jan 28, 2017 at 03:04:56PM +0100, Daniel Reichelt wrote:
>> I highly suspect this stems from packages' rules files supporting
>> reproducible builds.
> 
> I rather think this is due to binNMUs not modifying debian/changelog…
> (in the source package while it's modified in the binary packages…)
> 

Makes sense. Thanks for the clarification, Holger.




signature.asc
Description: OpenPGP digital signature

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Holger Levsen]

On Sat, Jan 28, 2017 at 03:04:56PM +0100, Daniel Reichelt wrote:
> I highly suspect this stems from packages' rules files supporting
> reproducible builds.

I rather think this is due to binNMUs not modifying debian/changelog…
(in the source package while it's modified in the binary packages…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Daniel Reichelt]

Hi,

I highly suspect this stems from packages' rules files supporting
reproducible builds.

The only way I see to solve this would be for the "reproducible builds"
infrastructure to hard-wire new timestamps at release-time of a new
package version.

Also: this is not limited to rsync. Basically any tool relying on
(mtime/file size) as a changed indicator is affected by this. Even if
the tool in question relied on (mtime/file size/inode number), "changed
checks" could be subverted in situations where changes are made to the
file directly instead of writing a new file and moving it into place
(thus retaining the inode number).


Cheers
Daniel



signature.asc
Description: OpenPGP digital signature

Re: Some Debian package upgrades are corrupting rsync "quick check" backups

[Christoph Biedl]

Adam Warner wrote...

> Why is a 27 January recompilation of the source package purporting to
> have the same modification time as the original binary package
> distributed 16 days earlier?

Lemme guess: For the sake of reproducible builds, the timestamp of all
created files is set to the time of the latest changelog entry.

> I suggest an rsync --checksum test on your backups (using
> --itemize-changes and --dry-run to check for modifications without
> making changes to the destination).

This is an ugly situation. Always using --checksum brings a huge penalty
indeed. The rsync program could take the source file's ctime into
account but I doubt there's support for that.

To me it seems saner to change Debian's workflow. Adding a new changelog
entry for a binNMU would break a lot of things, though. The least
intrusive but quite hacky way I can think of was to modify the timestamp
by adding one second for the first binNMU and so on. Nothing that gets
implemented over night, though.

Christoph


signature.asc
Description: Digital signature

Some Debian package upgrades are corrupting rsync "quick check" backups

[Adam Warner]

Hi all,

rsync typically detects file modifications by comparing exact
modification time and size of each identically named file in the source
and destination locations.

An rsync backup can be surreptitiously corrupted by modifying a source
file's content while keeping its size and modification time the same.

If some program is doing this then one way for rsync to detect the
modification is by appending the --checksum option. This requires every
file in the source and destination to be fully read. This can be many
orders of magnitude slower than a "quick check".

If you have been using rsync without --checksum to back up your Debian
partitions then your backups are likely corrupted.

Some packages are being generated with files containing exactly the
same size and timestamps even though the contents of those files are
different. This is how the data corruption arises:

1. You use rsync to back up your OS partition.
2. You perform package upgrades. Some files are replaced with different
content but have the same size and modification time.
3. You use rsync to back up your OS partition to the same destination.
The modified files with the same size and modification time are
skipped. Your backup is now corrupt (you have old files mixed with new
files).

Here's a recent example of a package upgrade causing this:




If you're tracking unstable you may have recently upgraded from
libqt5concurrent5_5.7.1+dfsg-3_amd64.deb to
libqt5concurrent5_5.7.1+dfsg-3+b1_amd64.deb (a subsequent binary non-
maintainer upload).

Here are the contents of both packages:
$ dpkg -c libqt5concurrent5_5.7.1+dfsg-3_amd64.deb 
drwxr-xr-x root/root 0 2017-01-12 04:14 ./
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/x86_64-linux-gnu/
-rw-r--r-- root/root 27352 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7.1
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/doc/
drwxr-xr-x root/root 0 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/
-rw-r--r-- root/root  1196 2016-12-01 21:17 
./usr/share/doc/libqt5concurrent5/LGPL_EXCEPTION.txt
-rw-r--r-- root/root 18232 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/changelog.Debian.gz
-rw-r--r-- root/root  3792 2016-12-01 21:17 
./usr/share/doc/libqt5concurrent5/changelog.gz
-rw-r--r-- root/root103466 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/copyright
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/overrides/
-rw-r--r-- root/root   230 2017-01-12 04:14 
./usr/share/lintian/overrides/libqt5concurrent5
lrwxrwxrwx root/root 0 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5 -> libQt5Concurrent.so.5.7.1
lrwxrwxrwx root/root 0 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7 -> libQt5Concurrent.so.5.7.1

$ dpkg -c libqt5concurrent5_5.7.1+dfsg-3+b1_amd64.deb 
drwxr-xr-x root/root 0 2017-01-12 04:14 ./
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/lib/x86_64-linux-gnu/
-rw-r--r-- root/root 27352 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7.1
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/doc/
drwxr-xr-x root/root 0 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/
-rw-r--r-- root/root  1196 2016-12-01 21:17 
./usr/share/doc/libqt5concurrent5/LGPL_EXCEPTION.txt
-rw-r--r-- root/root   252 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/changelog.Debian.amd64.gz
-rw-r--r-- root/root 18232 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/changelog.Debian.gz
-rw-r--r-- root/root  3792 2016-12-01 21:17 
./usr/share/doc/libqt5concurrent5/changelog.gz
-rw-r--r-- root/root103466 2017-01-12 04:14 
./usr/share/doc/libqt5concurrent5/copyright
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/
drwxr-xr-x root/root 0 2017-01-12 04:14 ./usr/share/lintian/overrides/
-rw-r--r-- root/root   230 2017-01-12 04:14 
./usr/share/lintian/overrides/libqt5concurrent5
lrwxrwxrwx root/root 0 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5 -> libQt5Concurrent.so.5.7.1
lrwxrwxrwx root/root 0 2017-01-12 04:14 
./usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7 -> libQt5Concurrent.so.5.7.1

In particular notice the shared library
/usr/lib/x86_64-linux-gnu/libQt5Concurrent.so.5.7.1 has an identical
size and timestamp.

However:
(testing) $ sha256sum