Re: Reproducible Builds for recent Debian security updates
Hi, On Sat, Mar 30, 2024 at 03:30:57PM -0700, Vagrant Cascadian wrote: > On 2024-03-30, Vagrant Cascadian wrote: > > On 2024-03-30, Salvatore Bonaccorso wrote: > >> On Fri, Mar 29, 2024 at 07:38:35PM -0700, Vagrant Cascadian wrote: > >>> Philipp Kern asked about trying to do reproducible builds checks for > >>> recent security updates to try to gain confidence about Debian's buildd > >>> infrastructure, given that they run builds in sid chroots which may have > >>> used or built or run a vulnerable xz-utils... > > ... > >> There would be an upcoming (or actually postponed) util-linux update > >> as well. Could you as extra paranoia please verify these here as well > >> (I assume its enough for you that the source package is signed, I > >> stripped the signature from the changes): > >> > >> https://people.debian.org/~carnil/tmp/util-linux/ > > > > I don't see any source packages there, just .deb .changes and signed > > .buildinfo files! The signed .buildinfo files are great, but would > > definitely need the source code ... looks like the util-linux changes > > are in a git branch, but a signed .dsc would be nice just to be sure I > > am testing the same thing. That said, testing from git and getting > > bit-for-bit identical results ... would be confidence inspiring! > > Hmmm. Might just go for it, and if we have issues, maybe try to dig up > > the .dsc? :) > > Hah. Almost in the time it took me to wonder about git vs. .dsc builds, > even with some minor differences in the build-depends, managed a > bit-for-bit identical build of util-linux:amd64 and util-linux:all! > > Tarball of build logs and .buildinfo files: > > > https://people.debian.org/~vagrant/util-linux-2.38.1-5+deb12u1.verification.tar.zst Thanks a lot! Regards, Salvatore
Re: Reproducible Builds for recent Debian security updates
On 2024-03-29, Vagrant Cascadian wrote: > So far, I have not found any reproducibility issues; everything I tested > I was able to get to build bit-for-bit identical with what is in the > Debian archive. > > I only tested bookworm security updates (not bullseye) ... > Not yet finished building: > > openvswitch So, the builds of openvswitch failed in the test suite... ... I performed another build with tests disabled, and the amd64 packages were bit-for-bit identical, but one of the arch:all packages, "openvswitch-source" had an already known issue; embedded information (username, uid, group, gid, timestamp ...) in the included tarball. This matches the previous version tested in the reproducible builds test infrastructure: https://tests.reproducible-builds.org/debian/dbdtxt/bookworm/amd64/openvswitch_3.1.0-2.diffoscope.txt.gz This is an explanable issue and I would say does not indicate anything surprising or unexpected or malicious, just unfortunate that it is not bit-for-bit reproducible, as it actually requires analysis! The good news is that newer versions (~3.2.2+) in Debian trixie and unstable of "openvswitch-source" fix this by shipping the source in a directory rather than a tarball, which dpkg normalizes when generating the .deb. So at least for future versions this issue is already fixed. live well, vagrant signature.asc Description: PGP signature
Re: Reproducible Builds for recent Debian security updates
On 2024-03-30, Vagrant Cascadian wrote: > On 2024-03-30, Salvatore Bonaccorso wrote: >> On Fri, Mar 29, 2024 at 07:38:35PM -0700, Vagrant Cascadian wrote: >>> Philipp Kern asked about trying to do reproducible builds checks for >>> recent security updates to try to gain confidence about Debian's buildd >>> infrastructure, given that they run builds in sid chroots which may have >>> used or built or run a vulnerable xz-utils... > ... >> There would be an upcoming (or actually postponed) util-linux update >> as well. Could you as extra paranoia please verify these here as well >> (I assume its enough for you that the source package is signed, I >> stripped the signature from the changes): >> >> https://people.debian.org/~carnil/tmp/util-linux/ > > I don't see any source packages there, just .deb .changes and signed > .buildinfo files! The signed .buildinfo files are great, but would > definitely need the source code ... looks like the util-linux changes > are in a git branch, but a signed .dsc would be nice just to be sure I > am testing the same thing. That said, testing from git and getting > bit-for-bit identical results ... would be confidence inspiring! > Hmmm. Might just go for it, and if we have issues, maybe try to dig up > the .dsc? :) Hah. Almost in the time it took me to wonder about git vs. .dsc builds, even with some minor differences in the build-depends, managed a bit-for-bit identical build of util-linux:amd64 and util-linux:all! Tarball of build logs and .buildinfo files: https://people.debian.org/~vagrant/util-linux-2.38.1-5+deb12u1.verification.tar.zst live well, vagrant signature.asc Description: PGP signature
Re: Reproducible Builds for recent Debian security updates
Hi, On Sat, Mar 30, 2024 at 03:05:03PM -0700, Vagrant Cascadian wrote: > On 2024-03-30, Salvatore Bonaccorso wrote: > > On Fri, Mar 29, 2024 at 07:38:35PM -0700, Vagrant Cascadian wrote: > >> Philipp Kern asked about trying to do reproducible builds checks for > >> recent security updates to try to gain confidence about Debian's buildd > >> infrastructure, given that they run builds in sid chroots which may have > >> used or built or run a vulnerable xz-utils... > ... > > Thanks a lot for doing this verification work! > > It is such an obvious application for Reproducible Builds that many > people have worked on for many years. So... I daresay, my pleasure and > honor. :) > > > > There would be an upcoming (or actually postponed) util-linux update > > as well. Could you as extra paranoia please verify these here as well > > (I assume its enough for you that the source package is signed, I > > stripped the signature from the changes): > > > > https://people.debian.org/~carnil/tmp/util-linux/ > > I don't see any source packages there, just .deb .changes and signed > .buildinfo files! The signed .buildinfo files are great, but would > definitely need the source code ... looks like the util-linux changes > are in a git branch, but a signed .dsc would be nice just to be sure I > am testing the same thing. That said, testing from git and getting > bit-for-bit identical results ... would be confidence inspiring! > Hmmm. Might just go for it, and if we have issues, maybe try to dig up > the .dsc? :) Sorry that was my fault obviously. The orig,tar.xz debian.tar.xz and dsc files are now there as well. Regards, Salvatore
Re: Reproducible Builds for recent Debian security updates
On 2024-03-30, Salvatore Bonaccorso wrote: > On Fri, Mar 29, 2024 at 07:38:35PM -0700, Vagrant Cascadian wrote: >> Philipp Kern asked about trying to do reproducible builds checks for >> recent security updates to try to gain confidence about Debian's buildd >> infrastructure, given that they run builds in sid chroots which may have >> used or built or run a vulnerable xz-utils... ... > Thanks a lot for doing this verification work! It is such an obvious application for Reproducible Builds that many people have worked on for many years. So... I daresay, my pleasure and honor. :) > There would be an upcoming (or actually postponed) util-linux update > as well. Could you as extra paranoia please verify these here as well > (I assume its enough for you that the source package is signed, I > stripped the signature from the changes): > > https://people.debian.org/~carnil/tmp/util-linux/ I don't see any source packages there, just .deb .changes and signed .buildinfo files! The signed .buildinfo files are great, but would definitely need the source code ... looks like the util-linux changes are in a git branch, but a signed .dsc would be nice just to be sure I am testing the same thing. That said, testing from git and getting bit-for-bit identical results ... would be confidence inspiring! Hmmm. Might just go for it, and if we have issues, maybe try to dig up the .dsc? :) live well, vagrant signature.asc Description: PGP signature
Re: Reproducible Builds for recent Debian security updates
Hi Vagrant, On Fri, Mar 29, 2024 at 07:38:35PM -0700, Vagrant Cascadian wrote: > Philipp Kern asked about trying to do reproducible builds checks for > recent security updates to try to gain confidence about Debian's buildd > infrastructure, given that they run builds in sid chroots which may have > used or built or run a vulnerable xz-utils... > > So far, I have not found any reproducibility issues; everything I tested > I was able to get to build bit-for-bit identical with what is in the > Debian archive. > > I only tested bookworm security updates (not bullseye), and I tested the > xz-utils update now present in unstable, which took a little trial and > error to find the right snapshot! The build dependencies for Debian > bookworm (a.k.a. stable) were *much* easier to satisfy, as it is not a > moving target! > > > Debian bookworm security updates verified: > > cacti iwd libuv1 pdns-recursor samba composer fontforge knot-resolver > php-dompdf-svg-lib squid yard > > Not yet finished building: > > openvswitch > > Did not yet try some time and disk-intensive builds: > > chromium firefox-esr thunderbird > > Debian unstable updates verified: > > xz-utils > > > A tarball of build logs (including some failed builds) and .buildinfo > files is available at: > > https://people.debian.org/~vagrant/debian-security-rebuilds.tar.zst > > > Some caveats: > > Notably, xz-utils has a build dependency that pulls in xz-utils, and the > version used may have been a vulnerable version (partly vulnerable?), > 5.6.0-0.2. > > The machine where I ran the builds had done some builds using packages > from sid over the last couple months, so may have at some point run the > vulnerable xz-utils code, so is not absolutely cleanest of > checks... but is at least some sort of data point. > > The build environment used tarballs that had usrmerge applied (as it is > harder to not apply usrmerge these days), while the buildd > infrastructure chroots do not have usrmerge applied. But this did not > appear to cause significant problems, although pulled in a few more perl > dependencies! > > > I used sbuild with the --chroot-mode=unshare mode. For the xz-utils > build I used some of the ideas developed in an earlier verification > builds experiment: > > > https://salsa.debian.org/reproducible-builds/debian-verification-build-experiment/-/blob/e003ddf19de13db2d512c25417e4bec863c3a082/sbuild-wrap#L71 > > > Was great to try and apply Reproducible Builds to real-world uses! Thanks a lot for doing this verification work! There would be an upcoming (or actually postponed) util-linux update as well. Could you as extra paranoia please verify these here as well (I assume its enough for you that the source package is signed, I stripped the signature from the changes): https://people.debian.org/~carnil/tmp/util-linux/ Regards, Salvatore
Reproducible Builds for recent Debian security updates
Philipp Kern asked about trying to do reproducible builds checks for recent security updates to try to gain confidence about Debian's buildd infrastructure, given that they run builds in sid chroots which may have used or built or run a vulnerable xz-utils... So far, I have not found any reproducibility issues; everything I tested I was able to get to build bit-for-bit identical with what is in the Debian archive. I only tested bookworm security updates (not bullseye), and I tested the xz-utils update now present in unstable, which took a little trial and error to find the right snapshot! The build dependencies for Debian bookworm (a.k.a. stable) were *much* easier to satisfy, as it is not a moving target! Debian bookworm security updates verified: cacti iwd libuv1 pdns-recursor samba composer fontforge knot-resolver php-dompdf-svg-lib squid yard Not yet finished building: openvswitch Did not yet try some time and disk-intensive builds: chromium firefox-esr thunderbird Debian unstable updates verified: xz-utils A tarball of build logs (including some failed builds) and .buildinfo files is available at: https://people.debian.org/~vagrant/debian-security-rebuilds.tar.zst Some caveats: Notably, xz-utils has a build dependency that pulls in xz-utils, and the version used may have been a vulnerable version (partly vulnerable?), 5.6.0-0.2. The machine where I ran the builds had done some builds using packages from sid over the last couple months, so may have at some point run the vulnerable xz-utils code, so is not absolutely cleanest of checks... but is at least some sort of data point. The build environment used tarballs that had usrmerge applied (as it is harder to not apply usrmerge these days), while the buildd infrastructure chroots do not have usrmerge applied. But this did not appear to cause significant problems, although pulled in a few more perl dependencies! I used sbuild with the --chroot-mode=unshare mode. For the xz-utils build I used some of the ideas developed in an earlier verification builds experiment: https://salsa.debian.org/reproducible-builds/debian-verification-build-experiment/-/blob/e003ddf19de13db2d512c25417e4bec863c3a082/sbuild-wrap#L71 Was great to try and apply Reproducible Builds to real-world uses! live well, vagrant signature.asc Description: PGP signature
