Am Sonntag, den 28.12.2008, 14:12 +0000 schrieb Mick: > I am struggling to understand why this problem occurs: > > I scp a number of binary files (e.g. videos) from box A to box B. I have > bz2'd them beforehand. At box B I run bzip2 -tv file.bz2, which reports that > all is OK. > > Then I run bzip2 -dv file.bz2 but it tells me that some of the contents have > been corrupted. Fair enough, I run bzip2recover which runs its CRC checks > and does not report anything being wrong. So, I know run bzip2 -tv > rec00*file.bz2 and it finds a number of files with "data integrity (CRC) > error in data". > > I run bzip2recover on box A and scp afresh copies of the affected rec00* > files. Some of the rec00* files seem to be copied without further > corruption, but one or two report again corruption (on box B only). > > Unable to understand why this is so, I decide to use vanilla ftp to another > box (in a data center this time) to copy over the same files from box A. > Then run ftp get from box B. bzip2 -tv shows no more corruption (although > this is not universal - some files still show corruption when tested on box > B). > > My ISP applies traffic shaping (especially on the obscure ssh port that I > have > chosen for the transmission) so some packets may be dropped, but this does > not explain why ftp occasionally works.
> Can you explain why this seemingly random corruption occurs? Most likely a broken NIC or NIC-driver. I had this two times (once with a rented root server). Problems occurred only when connection speed was high. For further tests., try also using a slow connection (<1 mbit). bandwidth limiting is possible on the command line with scp and rsync. > Can I trust bzip2 and its CRC tests? I think you can, but _much_ better is using MD5 or SHAx checksums. Use: source:~> md5sum * > files.md5 (copy files and files.md5 over to dest.) destin:~> md5sum -c files.md5 Exchange md5sum for one of sha1sum, sha224sum, sha256sum, sha384sum, sha512sum. That's exactly what .deb, .rpm and gentoo does with their packages. Bye, Daniel
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
