Bug#788185: squashfs-tools: Occasional corruption of large files.

[Steve Newcomb]

The purpose of this note is to report more information about this bug.

(1) I suspect this is the same bug that I reported earlier in 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781894. 



That earlier report provides a link to some sample data which 
demonstrates that the same discrepancy *always* appears in the resulting 
squashed filesystem, regardless of the version of squashfs-tools used, 
the compressor specified, the number of processors specified (or any 
other command-line options that I have tried), and even the number of 
virtual or real processor cores present on the machine.


(2) I have confirmed that the bug still exists in the alpha 4.3-2 
version of squashfs-tools, which I downloaded from 
https://packages.debian.org/sid/squashfs-tools


(3) Since I still need some sort of work-around for the problem, I have 
tried, unsuccessfully, to confirm the result here described by


Andreas Krüger  



I understood (perhaps wrongly) that he reports or at least suspects that 
the discrepancy problem does not occur in the context of a 
single-processor machine or a single-processor VM.  However, using such 
processing contexts, my own results are that the same discrepancy always 
occurs regardless.  In one such trial, I used BIOS options to disable 
all but one core and hyperthreading within that core.  In another trial, 
I used a VirtualBox VM with access to only one core.  The result is 
always discrepant in the same manner, regardless.


Therefore, contrary to Andreas's conjecture, I'm seeing no evidence that 
the bug is due to a race condition between processes.  The discrepancy 
always looks like this:



original recovered from squashfs
contents differ:  byte # 6614941696  0x73 s  0x00
contents differ:  byte # 6614941697  0x74 t  0x00
contents differ:  byte # 6614941700  0x73 s  0x00
contents differ:  byte # 6614941701  0x65 e  0x00
contents differ:  byte # 6614941702  0x61 a  0x00

 etc ... the rest of the squashed file continues to be bytes of value 
0x00.


Steve Newcomb



On Tue, 09 Jun 2015 09:58:05 +0200 =?UTF-8?B?QW5kcmVhcyBLcsO8Z2Vy?= wrote:
> Package: squashfs-tools
> Version: 1:4.2+20130409-2
> Severity: normal
>
> Hello,
>
> I'm using squashfs for full backups of my laptop, from a (quiet)
> LVM snapshot. I run sha256sum checksums on each file, then run
>
> mksquashfs /freeze /backup/$timestamp.backup.squashfs \
> -always-use-fragments -processors 2
>
> then mount the resulting squashfs file system and compare the checksums.
>
> Over the years, I've seen checksum errors occasionally.
>
> I have recently started to use virtual machines, so I now have more
> very large files in my backup. Since then, the number of problems
> seemed to have increased substantially
>
> The current backup saw two of these. Interestingly, in both
> cases, the problem is very close to the end of the file. For the
> record: In either case, the file length, according to ls -l, is
> the same for the original and the copy in the squashfs.
>
> One file is 2776104960 bytes long, a plain "cmp" run finds the
> first inconsistency at byte 2776018945, a mere 86015 bytes from
> the end.
>
> The other file is 7831814144 bytes long, a plain "cmp" finds the
> first inconsistency at byte 7831683073, a mere 131071 bytes from
> the end.
>
> Regards, and thank you for providing fine software
>
> Andreas
>
>
> -- System Information:
> Debian Release: 8.1
> APT prefers stable-updates
> APT policy: (500, 'stable-updates'), (500, 'stable')
> Architecture: amd64 (x86_64)
> Foreign Architectures: i386
>
> Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
> Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
>
> Versions of packages squashfs-tools depends on:
> ii libc6 2.19-18
> ii liblzma5 5.1.1alpha+20120614-2+b3
> ii liblzo2-2 2.08-1.2
> ii zlib1g 1:1.2.8.dfsg-2+b1
>
> squashfs-tools recommends no packages.
>
> squashfs-tools suggests no packages.
>

Bug#788185: squashfs-tools: Occasional corruption of large files.

[Andreas Krüger]

Hello, Steve,

you
> understood ... that he [me] reports or at least suspects that the discrepancy
> problem does not occur in the context of a single-processor machine or a
> single-processor VM.
I did indeed speculate at one time the problem might be multi-processor related.
But attempts to verify this failed. So forget about that one.

Regards, Andreas



signature.asc
Description: OpenPGP digital signature

Bug#788185: squashfs-tools: Occasional corruption of large files.

[Andreas Krüger]

Package: squashfs-tools
Version: 1:4.2+20130409-2
Severity: normal

Hello,

I'm using squashfs for full backups of my laptop, from a (quiet)
LVM snapshot. I run sha256sum checksums on each file, then run

mksquashfs /freeze /backup/$timestamp.backup.squashfs \
  -always-use-fragments -processors 2

then mount the resulting squashfs file system and compare the checksums.

Over the years, I've seen checksum errors occasionally.

I have recently started to use virtual machines, so I now have more
very large files in my backup.  Since then, the number of problems
seemed to have increased substantially

The current backup saw two of these.  Interestingly, in both
cases, the problem is very close to the end of the file.  For the
record: In either case, the file length, according to ls -l, is
the same for the original and the copy in the squashfs.

One file is 2776104960 bytes long, a plain cmp run finds the
first inconsistency at byte 2776018945, a mere 86015 bytes from
the end.

The other file is 7831814144 bytes long, a plain cmp finds the
first inconsistency at byte 7831683073, a mere 131071 bytes from
the end.

Regards, and thank you for providing fine software

Andreas


-- System Information:
Debian Release: 8.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages squashfs-tools depends on:
ii  libc6  2.19-18
ii  liblzma5   5.1.1alpha+20120614-2+b3
ii  liblzo2-2  2.08-1.2
ii  zlib1g 1:1.2.8.dfsg-2+b1

squashfs-tools recommends no packages.

squashfs-tools suggests no packages.

-- no debconf information




signature.asc
Description: OpenPGP digital signature