Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On Mon, 2017-08-28 at 12:58 +0200, Aurelien Jarno wrote: [...] > > > These files haven't been built on a build daemon, but instead have > > > been uploaded by the maintainer [1]. This is therefore not a buildd > > > issue, the issue has been fixed there already with the upgrade to > > > stretch. > > > > More precisely the two latest changelog entries have the same date: > > I have just filled a bug against ftp.debian.org for such packages to be > rejected. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873489 > > Aurelien Thank you for your prompt and accurate analysis of the situation and your bug submission to the FTP Masters suggesting that Lintian is used to reject maintainer uploads with a non-subsequent changelog date. Regards, Adam
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On 2017-08-28 12:42, Aurelien Jarno wrote: > On 2017-08-28 12:33, Aurelien Jarno wrote: > > On 2017-08-28 18:06, Adam Warner wrote: > > > On Sat, 2017-05-13 at 22:48 +0200, Aurelien Jarno wrote: > > > > On 2017-05-13 21:34, Aurelien Jarno wrote: > > > > > On 2017-05-13 17:52, Mattia Rizzolo wrote: > > > > > > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > > > > > > a) Has anything changed in the meantime? > > > > > > > > > > > > Yes: sbuild stopped repeating the changelog time taking it from > > > > > > the last > > > > > > entry, and will instead generate a new timestamp based on the > > > > > > current > > > > > > time: > > > > > > > > > > > > * For binNMUs, instead of copying the timestamp of the last > > > > > > changelog entry, > > > > > > generate a new one (closes: #843773) > > > > > > > > > > > > In version 0.73.0-1. > > > > > > > > > > And I am glad that after all that months with people talking about > > > > > the > > > > > issue, I finally got a detailed description of the issue and a > > > > > pointer > > > > > to the commit to backport. I'll work on that. > > > > > > > > The above change should now be deployed on most jessie based buildds, > > > > it's only missing on the buildds that are currently down. > > > > > > Original thread author here reporting to beware that some rsync data > > > corruption can still become apparent after all this time. > > > > > > It's been a while since I did a full rsync checksum test. Decided to do > > > one after a recent upgrade that includes clang 3.9 related files. These > > > are Debian systems that default to unstable BUT include all debian apt > > > sources including experimental/unstable/testing/stable/oldstable. > > > > > > I found these four corrupted files in my rsync backups: > > > > > > var/lib/dpkg/info/clang-3.9.md5sums > > > var/lib/dpkg/info/libclang-common-3.9-dev.md5sums > > > var/lib/dpkg/info/libclang1-3.9:amd64.md5sums > > > var/lib/dpkg/info/libllvm3.9:amd64.md5sums > > > > > > The latest packages were installed from this repository: > > > > > > Get:17 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > > clang-3.9 amd64 1:3.9.1-11 [37.3 MB] > > > Get:18 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > > libclang-common-3.9-dev amd64 1:3.9.1-11 [2,587 kB] > > > Get:19 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > > libllvm3.9 amd64 1:3.9.1-11 [11.4 MB] > > > Get:20 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > > libclang1-3.9 amd64 1:3.9.1-11 [5,896 kB] > > > > > > > These files haven't been built on a build daemon, but instead have > > been uploaded by the maintainer [1]. This is therefore not a buildd > > issue, the issue has been fixed there already with the upgrade to > > stretch. > > More precisely the two latest changelog entries have the same date: > I have just filled a bug against ftp.debian.org for such packages to be rejected. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873489 Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On 2017-08-28 12:33, Aurelien Jarno wrote: > On 2017-08-28 18:06, Adam Warner wrote: > > On Sat, 2017-05-13 at 22:48 +0200, Aurelien Jarno wrote: > > > On 2017-05-13 21:34, Aurelien Jarno wrote: > > > > On 2017-05-13 17:52, Mattia Rizzolo wrote: > > > > > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > > > > > a) Has anything changed in the meantime? > > > > > > > > > > Yes: sbuild stopped repeating the changelog time taking it from > > > > > the last > > > > > entry, and will instead generate a new timestamp based on the > > > > > current > > > > > time: > > > > > > > > > > * For binNMUs, instead of copying the timestamp of the last > > > > > changelog entry, > > > > > generate a new one (closes: #843773) > > > > > > > > > > In version 0.73.0-1. > > > > > > > > And I am glad that after all that months with people talking about > > > > the > > > > issue, I finally got a detailed description of the issue and a > > > > pointer > > > > to the commit to backport. I'll work on that. > > > > > > The above change should now be deployed on most jessie based buildds, > > > it's only missing on the buildds that are currently down. > > > > Original thread author here reporting to beware that some rsync data > > corruption can still become apparent after all this time. > > > > It's been a while since I did a full rsync checksum test. Decided to do > > one after a recent upgrade that includes clang 3.9 related files. These > > are Debian systems that default to unstable BUT include all debian apt > > sources including experimental/unstable/testing/stable/oldstable. > > > > I found these four corrupted files in my rsync backups: > > > > var/lib/dpkg/info/clang-3.9.md5sums > > var/lib/dpkg/info/libclang-common-3.9-dev.md5sums > > var/lib/dpkg/info/libclang1-3.9:amd64.md5sums > > var/lib/dpkg/info/libllvm3.9:amd64.md5sums > > > > The latest packages were installed from this repository: > > > > Get:17 https://cdn-aws.deb.debian.org/debian unstable/main amd64 clang-3.9 > > amd64 1:3.9.1-11 [37.3 MB] > > Get:18 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > libclang-common-3.9-dev amd64 1:3.9.1-11 [2,587 kB] > > Get:19 https://cdn-aws.deb.debian.org/debian unstable/main amd64 libllvm3.9 > > amd64 1:3.9.1-11 [11.4 MB] > > Get:20 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > > libclang1-3.9 amd64 1:3.9.1-11 [5,896 kB] > > > > These files haven't been built on a build daemon, but instead have > been uploaded by the maintainer [1]. This is therefore not a buildd > issue, the issue has been fixed there already with the upgrade to > stretch. More precisely the two latest changelog entries have the same date: | llvm-toolchain-3.9 (1:3.9.1-11) unstable; urgency=medium | | [ Sylvestre Ledru ] | * Remove the --no-discard-stderr option from help2man calls | * Also add a missing include in ftfbs-gcc.diff to fix a ftbfs | with gcc 7 | * clang was producing unusable binaries on armv5tel (Closes #873304) | Thanks to Adrian Bunk for the patch | * Disable -gsplit-dwarf when using gcc 7 for causing a linking issue | See https://bugs.llvm.org/show_bug.cgi?id=34140 (Closes: #853524) | | [ Gianfranco Costamagna, John Paul Adrian Glaubitz ] | * Add powerpcspe to latomic archs | | [ Katsuhiko Nishimra ] | * Ensure /usr/bin/g++-$(GCC_VERSION) exists (Closes: #871591) | | -- Sylvestre Ledru Sun, 18 Jun 2017 19:12:15 +0200 | | llvm-toolchain-3.9 (1:3.9.1-10) unstable; urgency=medium | | * Now that strech has been released, upload in unstable! | This is necessary for rust in unstable | * Try to fix some PATH_MAX on hurd | * Enable the verbose mode when trying to build libfuzzer | to detect potential issues in the path search | | -- Sylvestre Ledru Sun, 18 Jun 2017 19:12:15 +0200 That's the reason why the files ended-up with the same date but different content. Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On 2017-08-28 18:06, Adam Warner wrote: > On Sat, 2017-05-13 at 22:48 +0200, Aurelien Jarno wrote: > > On 2017-05-13 21:34, Aurelien Jarno wrote: > > > On 2017-05-13 17:52, Mattia Rizzolo wrote: > > > > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > > > > a) Has anything changed in the meantime? > > > > > > > > Yes: sbuild stopped repeating the changelog time taking it from > > > > the last > > > > entry, and will instead generate a new timestamp based on the > > > > current > > > > time: > > > > > > > > * For binNMUs, instead of copying the timestamp of the last > > > > changelog entry, > > > > generate a new one (closes: #843773) > > > > > > > > In version 0.73.0-1. > > > > > > And I am glad that after all that months with people talking about > > > the > > > issue, I finally got a detailed description of the issue and a > > > pointer > > > to the commit to backport. I'll work on that. > > > > The above change should now be deployed on most jessie based buildds, > > it's only missing on the buildds that are currently down. > > Original thread author here reporting to beware that some rsync data > corruption can still become apparent after all this time. > > It's been a while since I did a full rsync checksum test. Decided to do > one after a recent upgrade that includes clang 3.9 related files. These > are Debian systems that default to unstable BUT include all debian apt > sources including experimental/unstable/testing/stable/oldstable. > > I found these four corrupted files in my rsync backups: > > var/lib/dpkg/info/clang-3.9.md5sums > var/lib/dpkg/info/libclang-common-3.9-dev.md5sums > var/lib/dpkg/info/libclang1-3.9:amd64.md5sums > var/lib/dpkg/info/libllvm3.9:amd64.md5sums > > The latest packages were installed from this repository: > > Get:17 https://cdn-aws.deb.debian.org/debian unstable/main amd64 clang-3.9 > amd64 1:3.9.1-11 [37.3 MB] > Get:18 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > libclang-common-3.9-dev amd64 1:3.9.1-11 [2,587 kB] > Get:19 https://cdn-aws.deb.debian.org/debian unstable/main amd64 libllvm3.9 > amd64 1:3.9.1-11 [11.4 MB] > Get:20 https://cdn-aws.deb.debian.org/debian unstable/main amd64 > libclang1-3.9 amd64 1:3.9.1-11 [5,896 kB] > These files haven't been built on a build daemon, but instead have been uploaded by the maintainer [1]. This is therefore not a buildd issue, the issue has been fixed there already with the upgrade to stretch. Aurelien [1] https://tracker.debian.org/news/866006 -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On Sat, 2017-05-13 at 22:48 +0200, Aurelien Jarno wrote: > On 2017-05-13 21:34, Aurelien Jarno wrote: > > On 2017-05-13 17:52, Mattia Rizzolo wrote: > > > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > > > a) Has anything changed in the meantime? > > > > > > Yes: sbuild stopped repeating the changelog time taking it from > > > the last > > > entry, and will instead generate a new timestamp based on the > > > current > > > time: > > > > > > * For binNMUs, instead of copying the timestamp of the last > > > changelog entry, > > > generate a new one (closes: #843773) > > > > > > In version 0.73.0-1. > > > > And I am glad that after all that months with people talking about > > the > > issue, I finally got a detailed description of the issue and a > > pointer > > to the commit to backport. I'll work on that. > > The above change should now be deployed on most jessie based buildds, > it's only missing on the buildds that are currently down. Original thread author here reporting to beware that some rsync data corruption can still become apparent after all this time. It's been a while since I did a full rsync checksum test. Decided to do one after a recent upgrade that includes clang 3.9 related files. These are Debian systems that default to unstable BUT include all debian apt sources including experimental/unstable/testing/stable/oldstable. I found these four corrupted files in my rsync backups: var/lib/dpkg/info/clang-3.9.md5sums var/lib/dpkg/info/libclang-common-3.9-dev.md5sums var/lib/dpkg/info/libclang1-3.9:amd64.md5sums var/lib/dpkg/info/libllvm3.9:amd64.md5sums The latest packages were installed from this repository: Get:17 https://cdn-aws.deb.debian.org/debian unstable/main amd64 clang-3.9 amd64 1:3.9.1-11 [37.3 MB] Get:18 https://cdn-aws.deb.debian.org/debian unstable/main amd64 libclang-common-3.9-dev amd64 1:3.9.1-11 [2,587 kB] Get:19 https://cdn-aws.deb.debian.org/debian unstable/main amd64 libllvm3.9 amd64 1:3.9.1-11 [11.4 MB] Get:20 https://cdn-aws.deb.debian.org/debian unstable/main amd64 libclang1-3.9 amd64 1:3.9.1-11 [5,896 kB] Here are the mtimes of the newly install files: # ls -la /var/lib/dpkg/info/clang-3.9.md5sums /var/lib/dpkg/info/libclang-common-3.9-dev.md5sums /var/lib/dpkg/info/libclang1-3.9:amd64.md5sums /var/lib/dpkg/info/libllvm3.9:amd64.md5sums -rw-r--r-- 1 root root 10855 Jun 19 05:12 /var/lib/dpkg/info/clang-3.9.md5sums -rw-r--r-- 1 root root 384 Jun 19 05:12 /var/lib/dpkg/info/libclang1-3.9:amd64.md5sums -rw-r--r-- 1 root root 18320 Jun 19 05:12 /var/lib/dpkg/info/libclang-common-3.9-dev.md5sums -rw-r--r-- 1 root root 371 Jun 19 05:12 /var/lib/dpkg/info/libllvm3.9:amd64.md5sums Here's a diff of /var/lib/dpkg/info/clang-3.9.md5sums (rsync backup vs current file): 2,11c2,11 < 3cbbedf0c71035cd3029dfe1b052e581 usr/lib/llvm-3.9/bin/c-index-test < 0fdde07673439d60670efed27456169a usr/lib/llvm-3.9/bin/clang < aa1260a7741de9cdf0c748f8b7462c1c usr/lib/llvm-3.9/bin/clang-apply-replacements < 996c76e1c1ab4075d016d4a860a616a2 usr/lib/llvm-3.9/bin/clang-check < 8021eba7391ad79567cf0d2cc71bacfa usr/lib/llvm-3.9/bin/clang-include-fixer < 048cc918522e9099d7085d734e7356cc usr/lib/llvm-3.9/bin/clang-query < 7dae8990cdbf679b19e370d62fea05e1 usr/lib/llvm-3.9/bin/clang-rename < 3f7c675b0be7c94c88e6b27f665236d2 usr/lib/llvm-3.9/bin/find-all-symbols < a2920ac65d5085773747078cdd099f87 usr/lib/llvm-3.9/bin/modularize < dd36711af0597df2c78f3583fb914728 usr/lib/llvm-3.9/bin/sancov --- > e48899b8ed6e12602edaec4be4cf5ba8 usr/lib/llvm-3.9/bin/c-index-test > c523712273fb46731e4ad88de23a00a6 usr/lib/llvm-3.9/bin/clang > 1d59562b0f0167b20b3f47af8da65bb2 > usr/lib/llvm-3.9/bin/clang-apply-replacements > de8b1949a36135e7f5eac0016e99c813 usr/lib/llvm-3.9/bin/clang-check > 458274598da467da470be2404374432e usr/lib/llvm-3.9/bin/clang-include-fixer > cca520673bf5586037e66b8384944576 usr/lib/llvm-3.9/bin/clang-query > 3d3b408633a6ffe416c28a281082ffde usr/lib/llvm-3.9/bin/clang-rename > 288d8965a7fa9389c37233d8751683a9 usr/lib/llvm-3.9/bin/find-all-symbols > ec0b482ab1af827ea958b51a87a096f8 usr/lib/llvm-3.9/bin/modularize > 21c34b02091b53c37a90306f1dea5d6a usr/lib/llvm-3.9/bin/sancov 109c109 < 32db7068b426f045c64d8642c1e6c022 usr/share/doc/clang-3.9/changelog.Debian.gz --- > 2ec59c559a5c304cacddea2f02d74697 usr/share/doc/clang-3.9/changelog.Debian.gz 116,123c116,123 < 3a8deaf272eff95b110efb05ebc9ee9d usr/share/man/man1/clang-apply-replacements-3.9.1.gz < efe297e129ca7d6823ac46f5b7a58644 usr/share/man/man1/clang-check-3.9.1.gz < 114d671d66872a188832cad20062cd7b usr/share/man/man1/clang-include-fixer-3.9.1.gz < 8992eb9feffb6b90c015a28785d0b33b usr/share/man/man1/clang-query-3.9.1.gz < c69a128b81404ecc2eac7e759608fd15 usr/share/man/man1/clang-rename-3.9.1.gz < 14f265858cf398647aa0be918e65dd6b usr/share/man/man1/find-all-symbols-3.9.1.gz < e54fd298f2fea34b272d44b7c8c372a2
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On Sat, May 13, 2017 at 10:48:18PM +0200, Aurelien Jarno wrote: > The above change should now be deployed on most jessie based buildds, > it's only missing on the buildds that are currently down. cool, thank you! -- cheers, Holger signature.asc Description: Digital signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On 2017-05-13 21:34, Aurelien Jarno wrote: > On 2017-05-13 17:52, Mattia Rizzolo wrote: > > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > > a) Has anything changed in the meantime? > > > > Yes: sbuild stopped repeating the changelog time taking it from the last > > entry, and will instead generate a new timestamp based on the current > > time: > > > > * For binNMUs, instead of copying the timestamp of the last changelog > > entry, > > generate a new one (closes: #843773) > > > > In version 0.73.0-1. > > And I am glad that after all that months with people talking about the > issue, I finally got a detailed description of the issue and a pointer > to the commit to backport. I'll work on that. The above change should now be deployed on most jessie based buildds, it's only missing on the buildds that are currently down. -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On 2017-05-13 17:52, Mattia Rizzolo wrote: > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > a) Has anything changed in the meantime? > > Yes: sbuild stopped repeating the changelog time taking it from the last > entry, and will instead generate a new timestamp based on the current > time: > > * For binNMUs, instead of copying the timestamp of the last changelog entry, > generate a new one (closes: #843773) > > In version 0.73.0-1. And I am glad that after all that months with people talking about the issue, I finally got a detailed description of the issue and a pointer to the commit to backport. I'll work on that. > > b) Will this affect stretch? If so, what do we need to do now? It depends by wat you mean by "affect stretch". According to the version above, the buildds running stretch are not affected by the bug. So it mostly depends where the packages are built, not for which distribution. Cheers, Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On Sat, May 13, 2017 at 05:52:04PM +0200, Mattia Rizzolo wrote: > On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > > a) Has anything changed in the meantime? > > Yes: sbuild stopped repeating the changelog time taking it from the last > entry, and will instead generate a new timestamp based on the current > time: > > * For binNMUs, instead of copying the timestamp of the last changelog entry, > generate a new one (closes: #843773) > > In version 0.73.0-1. this is correct, but AFAIK this hasnt been deployed on the buildd yet. I'd be glad to be corrected about this… > > b) Will this affect stretch? If so, what do we need to do now? > IMHO, nothing. we might need to reschedule some packages… -- cheers, Holger signature.asc Description: Digital signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
On Sat, May 13, 2017 at 03:44:57PM +0100, Chris Lamb wrote: > a) Has anything changed in the meantime? Yes: sbuild stopped repeating the changelog time taking it from the last entry, and will instead generate a new timestamp based on the current time: * For binNMUs, instead of copying the timestamp of the last changelog entry, generate a new one (closes: #843773) In version 0.73.0-1. > b) Will this affect stretch? If so, what do we need to do now? Yes. IMHO, nothing. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
Hi all, Just following up on this thread. As it's been a while, here's the thread index: https://lists.debian.org/debian-security/2017/01/threads.html#00014 ... but AIUI, the issue is as follows. Naturally, let me know if this a poor or inaccurate summary: * binNMUs do not bump debian/changelog. * This causes mtimes to to not be bumped in the (actually modified!) package. * Backup programs (eg. rsync) will therefore not believe a binary has changed and thus will be skipped over. * A binary restored from such a backup may not execute correctly. * Asking users to change anything at all is always problematic, but to ask them to switch to using (for example) --checksum with its non-trivial performance penalty is a very difficult ask indeed. Some backup programs may not even support such a mode anyway… My questions are as follows: a) Has anything changed in the meantime? b) Will this affect stretch? If so, what do we need to do now? Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Re: [buildd-tools-devel] Some Debian package upgrades are corrupting rsync "quick check" backups
Hi, Quoting Holger Levsen (2017-01-30 14:58:33) > On Mon, Jan 30, 2017 at 02:47:45PM +0100, Johannes Schauer wrote: > > > b.) if thats the case, shall we scan all packages in sid for files which > > > have the same timestamp+filename but different checksums and ask for > > > binNMUs of those packages? > > The version of sbuild used on the buildds probably doesn't bump the > > timestamp > > yet. At least the binNMU-ed packages on my system share the binNMU changelog > > timestamp with the timestamp of the last source upload. :) > > so it seems we should > > a.) get this fix deployed on all the buildds. (how?) > b.) do the b.) above… (because AIUI this issue atm is only be noticed > by those running unstable or stretch, while once stretch is released many > more people will notice it…) I'm not involved in buildd development. I think Aurélien Jarno (CC) is involved as there are many commits from him in the respective sbuild git branch. Possibly the right mailing is debian-dak@l.d.o but I'm not active there. The buildds are running a custom version of sbuild with their own patches on top. So I'd expect that once Stretch is released, they will rebase their patches on top of sbuild from Stretch. So as far as I understand Henrik's original message, their problem should be fixed once packages start being built with sbuild from Stretch which happens after the Stretch release. Thanks! cheers, josch signature.asc Description: signature
