Re: Debian Buster will only be 54% reproducible (while we could be at >90%)
On Sat, Mar 09, 2019 at 09:15:34PM +, Dimitri John Ledkov wrote: [packages uploaded before December 2016, which didnt see binNMU since then, dont have a .buildinfo file (and were built with a dpkg which produces unreproducible packages] > So I guess after we release buster, we should do a mass-source-nmu > no-change uploads to make the .deb reproducible as shipped in the > archive. maybe. I think I first want to have the numbers right (and then current) to see how many packages are affected. Also: a not so low percentages of packages is still uploaded as non-source-only upload. I *believe* this should be fine (as nowadays the upload contains the .buildinfo file of that developer build, so if a package is reproducible this .buildinfo file can be taken). > Is this thus then an effectively a maas bug-file request to > sourcefully rebuild packages? no, at least not yet. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Debian Buster will only be 54% reproducible (while we could be at >90%)
On Wed, Mar 06, 2019 at 09:40:23AM +, peter green wrote: > > Because of their design, binNMUs are unreproducible, see #894441 [3] for > > the details (in short: binNMUs are not what they are ment to be: the source > > is changed and thrown away) > To be specific, the source tree is extracted, then an entry is added to > debian/changelog and then the package is built. This modified source > tree is not retained. indeed. > It seems to me that binnmus could be made reproducible by storing the > debian/changelog modifications in the buildinfo, then re-applying it > at reproduction time. that's actually the case nowadays, not sure since when. eg https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/2019/04/23/dmtx-utils_0.7.6-1.1+b1_kfreebsd-amd64.buildinfo starts like this: Format: 1.0 Source: dmtx-utils (0.7.6-1.1) Binary: dmtx-utils Architecture: kfreebsd-amd64 Version: 0.7.6-1.1+b1 Binary-Only-Changes: dmtx-utils (0.7.6-1.1+b1) sid; urgency=low, binary-only=yes . * Binary-only non-maintainer upload for kfreebsd-amd64; no source changes. * rebuild for libdmtx0b . -- kfreebsd-amd64 / kfreebsd-i386 Build Daemon (kamp) Sun, 14 Apr 2019 01:03:43 + [...] I yet have to actually test if one can bit by bit reproduce binNMUs with this information, but I'm quite very hopeful. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Debian Buster will only be 54% reproducible (while we could be at >90%)
On Tue, 5 Mar 2019 at 13:34, Holger Levsen wrote: > > hi, > > disclaimer: this has not yet been verified by anyone other than myself, > so I could very well be wrong. Reproducible builds are about enabling > anyone to independently verify that... ;p > > > == Reproducibility in theory == > > According to > https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html > we have 26476 source packages (92.8%) which can be built reproducibly in > buster/amd64, out of 28523 source packages in total. > (These 28523 source packages build 57448 binary packages.) > > But these tests are done without looking at the actual .deb files distributed > from ftp.debian.org (and we always knew that and pointed it out: > "93% reproducible _in our current test framework_".) > So I guess after we release buster, we should do a mass-source-nmu no-change uploads to make the .deb reproducible as shipped in the archive. Is this thus then an effectively a maas bug-file request to sourcefully rebuild packages? -- Regards, Dimitri.
Re: Debian Buster will only be 54% reproducible (while we could be at >90%)
Am Mi., 6. März 2019 um 10:40 Uhr schrieb peter green : > > > Because of their design, binNMUs are unreproducible, see #894441 [3] for > > the details (in short: binNMUs are not what they are ment to be: the source > > is changed and thrown away) > To be specific, the source tree is extracted, then an entry is added to > debian/changelog and then the package is built. This modified source tree is > not retained. > [...] (Experience report incoming) I have once tried that in the Tanglu derivative, and found out that this wasn't as easy as I initially thought because a lot of packages run special tools prior to building their sources, e.g. to edit d/control or to read d/changelog and inject data in several places. So, the option there was either to create a chroot dedicated to the source package rebuild (installing all Build-Deps and Pre-Deps prior to the actual source rebuild), or to not actually rebuild the source package but just edit the d/changelog file and recreate the tarball. For Tanglu we went for the "just edit d/changelog and re-tar, re-sign & upload" which worked fine and without any noticeable issues - this was mainly due to the limited build power we had at the time. For Debian, which has a lot more resources, just full rebuilding the source with all dependencies is likely much cleaner, but this approach might be a bit slow for huge transitions. At Ubuntu, some people seem to do this process manually, that is run some scripts locally, rebuild sources locally & upload (unless that has changed recently). In general, having source d/changelog aligned with the actual binaries produced is a really great goal! Cheers, Matthias -- I welcome VSRE emails. See http://vsre.info/
re: Debian Buster will only be 54% reproducible (while we could be at >90%)
Because of their design, binNMUs are unreproducible, see #894441 [3] for the details (in short: binNMUs are not what they are ment to be: the source is changed and thrown away) To be specific, the source tree is extracted, then an entry is added to debian/changelog and then the package is built. This modified source tree is not retained. It seems to me that binnmus could be made reproducible by storing the debian/changelog modifications in the buildinfo, then re-applying it at reproduction time.
Debian Buster will only be 54% reproducible (while we could be at >90%)
hi, disclaimer: this has not yet been verified by anyone other than myself, so I could very well be wrong. Reproducible builds are about enabling anyone to independently verify that... ;p == Reproducibility in theory == According to https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html we have 26476 source packages (92.8%) which can be built reproducibly in buster/amd64, out of 28523 source packages in total. (These 28523 source packages build 57448 binary packages.) But these tests are done without looking at the actual .deb files distributed from ftp.debian.org (and we always knew that and pointed it out: "93% reproducible _in our current test framework_".) == Looking at binary packages Debian actually distributes == So, Vagrant came up with an idea [1] to check buildinfo.debian.net for .deb files for which 2 or more .buildinfo exist (where "exist" means that the .deb files sha1sum is listed in the .buildinfo file) and I turned that into a jenkins job doing this check for all 57448 binary packages in amd64/buster/main (incl downloading all those .deb files from ftp.d.o). The current main results (from this job [2]) are: reproducible packages in buster/amd64: 30885: (53.7600%) unreproducible packages in buster/amd64: 26543: (46.2000%) and reproducible binNMUs in buster/amd64: 0: (0%) unreproducible binNMU in buster/amd64: 7423: (12.9200%) == why are binNMUs unreproducible? == Because of their design, binNMUs are unreproducible, see #894441 [3] for the details (in short: binNMUs are not what they are ment to be: the source is changed and thrown away) and our proposed solution: 'binNMUs should be replaced by easy "no-change-except-debian/changelog-uploads'. So that accounts for 12%, but 12% are not enough to explain the difference between 54% and 93%... == packages which have not been rebuilt since December 2016 == And today I remember a thread I started last year in May, titled "packages which have not been rebuilt since December 2016" [4] (because these packages were build with an old dpkg not producing .buildinfo files, which Chris turned into #900837 [5] "release.debian.org: Mass-rebuild of packages for reproducible builds" and so today I ran Chris' script [6] again on coccia.d.o, and today it showed that 'only' 6804 source packages need a rebuild (compared to 9192 eight months ago). 6804 of of 28523 is 23.9%. And 54%+12%+24% equals 90%. Bingo. Bummer. (While #900837 was only filed in 2018 we knew about this issue since 2015 or so... probably earlier. Sigh.) == After the release is before the release. == So, as we first need to fix #894441 before we can sensibly fix #900837 and because Buster is practically frozen, I think we can just conclude that Buster is quite reproducible in theory (similar but better than Stretch...) and that we need to make sure to address #894441 ASAP, which means for Bullseye, the release after Buster. Fur future reference, a summary of the current status of Debian's reproducibiliy is available at https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues [7] Happy hacking and many many thanks to everyone who has contributed so far! [1] https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001239.html [2] https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/103/console [3] https://bugs.debian.org/894441 [4] https://lists.debian.org/debian-devel/2018/05/msg00499.html [5] https://bugs.debian.org/900837 [6] https://lists.debian.org/debian-devel/2018/06/msg7.html [7] https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
