Bug#774415: [buildd-tools-devel] Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, On Wed, Dec 21, 2016 at 10:52:54AM +0100, Johannes Schauer wrote: > because somebody who *does* care about this and uses their own local mirror > will have their setup subverted by this feature without any warning. well, I (the user) feed it with a .buildinfo files, which needs old versions. So I basically *ask* for snapshot.d.o to be used. I'd *expect* that snapshot.d.o is being used! > > why? If I call "sbuild foo.buildinfo" and that .buildinfo file has no > > source hashes, then sbuild should fail. Easy. (?!?!) > > Why should it then fail in your opinion? because for rebuilding purposes, .buildinfo files without source hashes are broken. (And as .buildinfo files are made for rebuilding, I'd say .buildinfo files without source hashes are always broken.) > Sure, it's easy to implement but I wonder if this restrictions makes sense. > Why > do you think it does? Because I dont expect there are .buildinfo files without source hashes. > > You seem to imply that the Debian autobuilders will generate .buildinfo > > files > > without source hashes - I think *that* is a problem - how can we fix it? > > Autobuilders only generate the arch:all and arch:any binary packages from the > source package they are given. They do not regenerate the source package. > Thus, > they will call dpkg-buildpackage with --build=any or --build=all which in turn > will create a .buildinfo that doesn't contain the source hash. SIGH. This is a *major* problem, I think. > If one tries passing --buildinfo-option=--build=full to dpkg-buildpackage then > this will lead to a build failure if dpkg-buildpackage was not also called > with > --build=full. This makes sense on the level of dpkg-buildpackage because it's > possible to build binary packages without having the source package. But on > the > autobuilder level the source package always exists. It would thus probably > have > to be sbuilds job to mangle the buildinfo file and insert the source package > hash in it. But if you do that then you get to disparities between people > generating their buildinfo with sbuild/pbuilder and people who just use > dpkg-buildpackage... makes sense and sucks. Need to think more about this. -- cheers, Holger signature.asc Description: Digital signature
Bug#774415: [buildd-tools-devel] Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Holger Levsen (2016-12-21 10:32:07) > On Wed, Dec 21, 2016 at 08:14:23AM +0100, Johannes Schauer wrote: > > > (though I'm not sure I fully understand why not assume -rb if > > > foo.buildinfo is given - I do understand for foo.changes…) > > > > - Because I'm not so sure that the user is aware that passing a .buildinfo > >file will mean that sbuild is querying snapshot.d.n without asking the > > user > >for further consent. > > what's the problem with that? (especially compared to downloading > packages from ftp.*.debian.org, which is also done…) because somebody who *does* care about this and uses their own local mirror will have their setup subverted by this feature without any warning. > > - Because then we would only allow .buildinfo files that include the > > source package hash as well which I find quite limiting - especially > > considering how the Debian autobuilders will exclusively generate > > .buildinfo files of that kind > > why? If I call "sbuild foo.buildinfo" and that .buildinfo file has no > source hashes, then sbuild should fail. Easy. (?!?!) Why should it then fail in your opinion? Sure, it's easy to implement but I wonder if this restrictions makes sense. Why do you think it does? > You seem to imply that the Debian autobuilders will generate .buildinfo files > without source hashes - I think *that* is a problem - how can we fix it? Autobuilders only generate the arch:all and arch:any binary packages from the source package they are given. They do not regenerate the source package. Thus, they will call dpkg-buildpackage with --build=any or --build=all which in turn will create a .buildinfo that doesn't contain the source hash. If one tries passing --buildinfo-option=--build=full to dpkg-buildpackage then this will lead to a build failure if dpkg-buildpackage was not also called with --build=full. This makes sense on the level of dpkg-buildpackage because it's possible to build binary packages without having the source package. But on the autobuilder level the source package always exists. It would thus probably have to be sbuilds job to mangle the buildinfo file and insert the source package hash in it. But if you do that then you get to disparities between people generating their buildinfo with sbuild/pbuilder and people who just use dpkg-buildpackage... cheers, josch signature.asc Description: signature
Bug#774415: [buildd-tools-devel] Bug#774415: From srebuild sbuild-wrapper to debrebuild
On Wed, Dec 21, 2016 at 08:14:23AM +0100, Johannes Schauer wrote: > > as long as there's a short version of --rebuild-and-verify=foo.buildinfo > > such as (for example) -rb=foo.buildinfo fine with me… :) > the -r short option is still free. sounds good. > > (though I'm not sure I fully understand why not assume -rb if foo.buildinfo > > is given - I do understand for foo.changes…) > > - Because I'm not so sure that the user is aware that passing a .buildinfo >file will mean that sbuild is querying snapshot.d.n without asking the user >for further consent. what's the problem with that? (especially compared to downloading packages from ftp.*.debian.org, which is also done…) > - Because then we would only allow .buildinfo files that include the source >package hash as well which I find quite limiting - especially considering >how the Debian autobuilders will exclusively generate .buildinfo files of >that kind why? If I call "sbuild foo.buildinfo" and that .buildinfo file has no source hashes, then sbuild should fail. Easy. (?!?!) You seem to imply that the Debian autobuilders will generate .buildinfo files without source hashes - I think *that* is a problem - how can we fix it? -- cheers, Holger signature.asc Description: Digital signature
Bug#774415: [buildd-tools-devel] Bug#774415: From srebuild sbuild-wrapper to debrebuild
Quoting Holger Levsen (2016-12-21 02:08:57) > On Wed, Dec 21, 2016 at 01:50:11AM +0100, Johannes Schauer wrote: > > Given all these arguments, adding a --rebuild-and-verify=foo.buildinfo > > option > > to sbuild sounds like the most sane thing to do. It would even not require > > the > > existing interface to change (the positional argument is a single source > > package). > > > > Any thoughts? > > as long as there's a short version of --rebuild-and-verify=foo.buildinfo > such as (for example) -rb=foo.buildinfo fine with me… :) the -r short option is still free. > (though I'm not sure I fully understand why not assume -rb if foo.buildinfo > is given - I do understand for foo.changes…) - Because I'm not so sure that the user is aware that passing a .buildinfo file will mean that sbuild is querying snapshot.d.n without asking the user for further consent. - Because then we would only allow .buildinfo files that include the source package hash as well which I find quite limiting - especially considering how the Debian autobuilders will exclusively generate .buildinfo files of that kind Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
On Wed, Dec 21, 2016 at 01:50:11AM +0100, Johannes Schauer wrote: > Given all these arguments, adding a --rebuild-and-verify=foo.buildinfo option > to sbuild sounds like the most sane thing to do. It would even not require the > existing interface to change (the positional argument is a single source > package). > > Any thoughts? as long as there's a short version of --rebuild-and-verify=foo.buildinfo such as (for example) -rb=foo.buildinfo fine with me… :) (though I'm not sure I fully understand why not assume -rb if foo.buildinfo is given - I do understand for foo.changes…) -- cheers, Holger signature.asc Description: Digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Holger Levsen (2016-12-19 10:35:53) > On Mon, Dec 19, 2016 at 10:02:58AM +0100, Johannes Schauer wrote: > > Other ways to solve this problem include: > > - only accept .buildinfo files that include the .dsc filename and checksum > > - accept .changes files that reference both the .buildinfo and the .dsc > > those two seem sensible to me. I see why you think those make sense and I see how I could agree with you. But thinking about this again, I think that the problem with implementing these two options would be that they would sneak in an unexpected privacy breach. If I add an option like --rebuild-and-verify=foo.buildinfo then the documentation for that command line option can include a big fat warning that using this option will try accessing snapshot.d.o to retrieve the right package versions. Including this warning if the buildinfo is used implicitly (for example if only a .changes file is passed) is not so easy. Furthermore, .changes files will now nearly always include a reference to the .buildinfo file. So if a .changes file were used, then another command line argument would be needed to turn the buildinfo verification on or off (depending on what the default is). Lastly, accepting bare .buildinfo files requires them to reference the .dsc which I find quite limiting. Builders should be able to verify .buildinfo files that do not contain the source package. Given all these arguments, adding a --rebuild-and-verify=foo.buildinfo option to sbuild sounds like the most sane thing to do. It would even not require the existing interface to change (the positional argument is a single source package). Any thoughts? cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Johannes Schauer (2016-12-20 13:49:27) > Currently, a buildinfo file does not specify which artifacts were supposed to > be built (source,any,all). as guillem points out to me on #debian-dpkg, the Architecture field lists exactly that. It will contain "source" if the source package was built, "all" for arch:all packages and a Debian architecture for arch:any builds. > What should happen if the buildinfo file was for an Arch:any build but when > rebuilding the source package, also arch:all packages show up? What should > happen if the original build was including the source package but now it does > not? > > Should the verification fail if the build produces artifacts that are not > listed in the buildinfo? > > Should the verification fail if the build produces does not produce artifacts > that are listed in the buildinfo? I thus propose that a builder should use the Architecture field to figure out what to build and fail if the produced artifacts are any different from the ones in the buildinfo (being as strict about it as possible). I implemented parsing of the Architecture field in the debrebuild.pl script and let it pass the --build, --host, --arch-all, --arch-any and --source options to sbuild. Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Johannes Schauer (2016-12-19 10:02:58) > I also came up with another question: and as I'm implementing this, here yet another: Currently, a buildinfo file does not specify which artifacts were supposed to be built (source,any,all). What should happen if the buildinfo file was for an Arch:any build but when rebuilding the source package, also arch:all packages show up? What should happen if the original build was including the source package but now it does not? Should the verification fail if the build produces artifacts that are not listed in the buildinfo? Should the verification fail if the build produces does not produce artifacts that are listed in the buildinfo? How should a rebuilder infer the value of the --build argument that it passes to dpkg-buildpackage to reproduce the given buildinfo file? Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
On Mon, Dec 19, 2016 at 10:02:58AM +0100, Johannes Schauer wrote: > Other ways to solve this problem include: > - only accept .buildinfo files that include the .dsc filename and checksum > - accept .changes files that reference both the .buildinfo and the .dsc those two seem sensible to me. -- cheers, Holger signature.asc Description: Digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting HW42 (2016-12-19 07:37:00) > So you (at least josch and ntyi) seem to prefer to have the user facing part > in sbuild/pbuilder and the common functionality in some kind of library. How > should the "library" interface look like for sbuild? pbuilder is written in bash, so a plain-text interface via stdin/stdout would be preferrable, I guess. The tool figuring out the right snapshot timestamps could just write an apt sources.list in deb822 format on standard output. Even if the builder doesn't use the apt resolver, the deb822 format is easy to parse and thus any required information can be extracted from it. I also came up with another question: sbuild/pbuilder/debrebuild are given a .buildinfo file. But that file does not necessarily include a reference to a .dsc. What shall be done in that case? The current debrebuild script will auto-generate the filename of the .dsc from the Source and Version fields of the .buildinfo file. Though the resulting .dsc might accidentally come from a different build. Is this okay? Other ways to solve this problem include: - only accept .buildinfo files that include the .dsc filename and checksum $ builder foo.buildinfo E: foo.buildinfo does not reference a source package - accept .changes files that reference both the .buildinfo and the .dsc $ builder foo.changes I: .changes file references a .buildinfo, will verify checksums at the end - make .buildinfo files a second-class citizen which are passed *in addition* to a .dsc $ builder foo.dsc --use-buildinfo=foo.buildinfo - accept .buildinfo files, autogenerate the .dsc filename but allow to override it via the command line $ builder foo.buildinfo --use-dsc=foo.dsc Thoughts? Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Johannes Schauer: > Hi, > > Quoting Niko Tyni (2016-12-17 13:40:32) >> On Thu, Dec 15, 2016 at 02:21:49PM +0100, Johannes Schauer wrote: >>> Quoting Niko Tyni (2016-12-15 14:04:19) In general, I like the concept of sbuild/pbuilder accepting .buildinfo files as input. This makes the user interface simple. My expectation for this mode of operation would be for the builder to recreate the old build as closely as possible. >>> >>> I agree but would add that they should also have the ability to tell the >>> user >>> if the checksums of the re-compiled packages do or do not match the >>> information >>> in the supplied .buildinfo file. >> >> I suppose; please just make sure such a failure is easily distinguishable >> from a failing build. > > My plan would be to add it as a success/failure line next to the lintian or > autopkgtest status at the bottom of the build log. > >>> I don't care whether we have debrebuild as a wrapper to sbuild/pbuilder or >>> sbuild/pbuilder use a common tool to figure out the right lines for the >>> sources.list inside the chroot. I just want to have .buildinfo support for >>> sbuild in Stretch and if either solution is not in unstable soon, then my >>> plan is to just add .buildinfo support to sbuild myself so that it's still >>> included in the next Debian stable release. >> >> Having this just inside sbuild for stretch and refactoring it out later >> if necessary works for me, but I'm not sure if HW42 and/or Mattia have >> thoughts about the pbuilder side? > > Putting them back in CC. > > I am especially waiting for a response from HW42 who volunteered to "keep an > eye on it" but who didn't come back to my pings on IRC yet. > > HW42: I need to know what your plan is for Stretch so that I can decide what > to > include in the next sbuild release. Sorry about the late reply. I didn't had any plans sofar for stretch. Given that a) the .buildinfo format it self b) the services around .buildinfo c) the interface of debrebuild (or buildinfo-utils, or whatever) is not really clear/finished yet I would expect that one need anyway the backports version. If you think otherwise we can of course push to get the current version into stretch. >> I note that we're only getting started on working with .buildinfo files. It >> seems possible that we encounter enough common tasks that something like a >> 'buildinfo-utils' package will be warranted, in which case a 'buildinfo >> find-debs' command would fit in there. > > I'm all in for breaking out common functionality into tools used by multiple > parties. So you (at least josch and ntyi) seem to prefer to have the user facing part in sbuild/pbuilder and the common functionality in some kind of library. How should the "library" interface look like for sbuild? signature.asc Description: OpenPGP digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Niko Tyni (2016-12-17 13:40:32) > On Thu, Dec 15, 2016 at 02:21:49PM +0100, Johannes Schauer wrote: > > Quoting Niko Tyni (2016-12-15 14:04:19) > > > In general, I like the concept of sbuild/pbuilder accepting .buildinfo > > > files > > > as input. This makes the user interface simple. My expectation for this > > > mode > > > of operation would be for the builder to recreate the old build as > > > closely as > > > possible. > > > > I agree but would add that they should also have the ability to tell the > > user > > if the checksums of the re-compiled packages do or do not match the > > information > > in the supplied .buildinfo file. > > I suppose; please just make sure such a failure is easily distinguishable > from a failing build. My plan would be to add it as a success/failure line next to the lintian or autopkgtest status at the bottom of the build log. > > I don't care whether we have debrebuild as a wrapper to sbuild/pbuilder or > > sbuild/pbuilder use a common tool to figure out the right lines for the > > sources.list inside the chroot. I just want to have .buildinfo support for > > sbuild in Stretch and if either solution is not in unstable soon, then my > > plan is to just add .buildinfo support to sbuild myself so that it's still > > included in the next Debian stable release. > > Having this just inside sbuild for stretch and refactoring it out later > if necessary works for me, but I'm not sure if HW42 and/or Mattia have > thoughts about the pbuilder side? Putting them back in CC. I am especially waiting for a response from HW42 who volunteered to "keep an eye on it" but who didn't come back to my pings on IRC yet. HW42: I need to know what your plan is for Stretch so that I can decide what to include in the next sbuild release. > I note that we're only getting started on working with .buildinfo files. It > seems possible that we encounter enough common tasks that something like a > 'buildinfo-utils' package will be warranted, in which case a 'buildinfo > find-debs' command would fit in there. I'm all in for breaking out common functionality into tools used by multiple parties. Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
On Thu, Dec 15, 2016 at 02:21:49PM +0100, Johannes Schauer wrote: > Quoting Niko Tyni (2016-12-15 14:04:19) > > In general, I like the concept of sbuild/pbuilder accepting .buildinfo files > > as input. This makes the user interface simple. My expectation for this mode > > of operation would be for the builder to recreate the old build as closely > > as > > possible. > > I agree but would add that they should also have the ability to tell the user > if the checksums of the re-compiled packages do or do not match the > information > in the supplied .buildinfo file. I suppose; please just make sure such a failure is easily distinguishable from a failing build. > > The other somewhat generic thing that the current script does / could > > do is mangling the package build dependencies to correspond to the > > exact versions in the .buildinfo file. This is a much easier and quite > > mechanical task, but it might still be worth to make it into a filter > > that takes one .dsc and outputs a modified one with strictly versioned > > build dependencies? > Doing the latter is extremely trivial: Right, I see. > I don't care whether we have debrebuild as a wrapper to sbuild/pbuilder or > sbuild/pbuilder use a common tool to figure out the right lines for the > sources.list inside the chroot. I just want to have .buildinfo support for > sbuild in Stretch and if either solution is not in unstable soon, then my plan > is to just add .buildinfo support to sbuild myself so that it's still included > in the next Debian stable release. Having this just inside sbuild for stretch and refactoring it out later if necessary works for me, but I'm not sure if HW42 and/or Mattia have thoughts about the pbuilder side? I note that we're only getting started on working with .buildinfo files. It seems possible that we encounter enough common tasks that something like a 'buildinfo-utils' package will be warranted, in which case a 'buildinfo find-debs' command would fit in there. (Another task for buildinfo-utils I can think of is diffing/intersecting build environment information in two .buildinfo files to determine different/common things about two builds, but this is getting off-topic for this bug...) -- Niko Tyni nt...@debian.org
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting Niko Tyni (2016-12-15 14:04:19) > > But then on IRC, HW42 suggested to approach this problem differently. > > Instead of integrating the functionality of figuring out the right > > repositories to reproduce the contents of a buildinfo file into sbuild, > > write a tool that can drive any package builder (like pbuilder). > > Thanks for your work. I had a belated look and I'm willing to (help to) > maintain this stuff in the reproducible-builds team. thanks! > In general, I like the concept of sbuild/pbuilder accepting .buildinfo files > as input. This makes the user interface simple. My expectation for this mode > of operation would be for the builder to recreate the old build as closely as > possible. I agree but would add that they should also have the ability to tell the user if the checksums of the re-compiled packages do or do not match the information in the supplied .buildinfo file. > Like you two, I'm also wondering about the interaction of the various tools > here. The heavy weight lifted by this one is locating the package versions > specified in a .buildinfo file. It seems to me that this functionality could > also be a standalone helper tool that sbuild and pbuilder could call to > determine the necessary apt sources.list entries. > > The other somewhat generic thing that the current script does / could > do is mangling the package build dependencies to correspond to the > exact versions in the .buildinfo file. This is a much easier and quite > mechanical task, but it might still be worth to make it into a filter > that takes one .dsc and outputs a modified one with strictly versioned > build dependencies? Doing the latter is extremely trivial: - one call to Dpkg::Control->new() to parse the .buildinfo - one call to Dpkg::Deps::deps_parse() to parse Installed-Build-Depends - one loop over the result, printing it The first step will even already be done in sbuild/pbuilder if they are the ones accepting .builidinfo files. So this doesn't seem complicated enough to warrant its own tool but I'm of course not stopping anybody from writing one. :) > With those, I'm thinking that 'sbuild foo.buildinfo' would fork out to > buildinfo-findrepos (or whatever) and possibly buildinfo-mangledsc, and then > read the environment variables, possible binNMU information etc. from the > .buildinfo file by itself. > > I suppose there could still be a 'debrebuild' or 'buildinfo-rebuild' > tool that just calls a builder with a .buildinfo file and compares the > resulting binaries. If sbuild/pbuilder are parsing the .buildinfo themselves, then they can compare the resulting binaries themselves as well. I think doing that would be expected when providing a .buildinfo as input to either. Then no further wrapper would be needed. I don't care whether we have debrebuild as a wrapper to sbuild/pbuilder or sbuild/pbuilder use a common tool to figure out the right lines for the sources.list inside the chroot. I just want to have .buildinfo support for sbuild in Stretch and if either solution is not in unstable soon, then my plan is to just add .buildinfo support to sbuild myself so that it's still included in the next Debian stable release. Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
On Tue, Aug 02, 2016 at 10:49:00PM +0200, Johannes Schauer wrote: > But then on IRC, HW42 suggested to approach this problem differently. Instead > of integrating the functionality of figuring out the right repositories to > reproduce the contents of a buildinfo file into sbuild, write a tool that can > drive any package builder (like pbuilder). Thanks for your work. I had a belated look and I'm willing to (help to) maintain this stuff in the reproducible-builds team. In general, I like the concept of sbuild/pbuilder accepting .buildinfo files as input. This makes the user interface simple. My expectation for this mode of operation would be for the builder to recreate the old build as closely as possible. Like you two, I'm also wondering about the interaction of the various tools here. The heavy weight lifted by this one is locating the package versions specified in a .buildinfo file. It seems to me that this functionality could also be a standalone helper tool that sbuild and pbuilder could call to determine the necessary apt sources.list entries. The other somewhat generic thing that the current script does / could do is mangling the package build dependencies to correspond to the exact versions in the .buildinfo file. This is a much easier and quite mechanical task, but it might still be worth to make it into a filter that takes one .dsc and outputs a modified one with strictly versioned build dependencies? With those, I'm thinking that 'sbuild foo.buildinfo' would fork out to buildinfo-findrepos (or whatever) and possibly buildinfo-mangledsc, and then read the environment variables, possible binNMU information etc. from the .buildinfo file by itself. I suppose there could still be a 'debrebuild' or 'buildinfo-rebuild' tool that just calls a builder with a .buildinfo file and compares the resulting binaries. Thoughts? I've pushed the current implementation (with a timestamp sorting fix) to https://anonscm.debian.org/cgit/reproducible/debrebuild.git/ but of course we can still generalize on that and change names etc. as necessary. -- Niko Tyni nt...@debian.org
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, Quoting HW42 (2016-11-17 05:10:00) > After discussing this in the irc meeting yesterday I propose that: > > - we keep it as a separate tool. > - put it in a git repo under >https://anonscm.debian.org/git/reproducible/ > - We have more than enough DDs who are willing to sponsor uploads, so >having it in the Debian archive is no problem. > - we mainly maintain this as a group. I will try to especially keep an >eye on it. > > Since you have done all the work so far the final decision is obviously > up to you. > > If the above is fine with you I will prepare packaging it during the next > week (I also have a few improvements planed). please go ahead. I'll make sure that sbuild adds a facility to receive a full binNMU changelog entry from the user. Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Johannes Schauer: > Hi, > > On Thu, 10 Nov 2016 05:54:13 -0200 Johannes Schauer wrote: >> On Tue, 02 Aug 2016 22:49:00 +0200 Johannes Schauer wrote: >>> But then on IRC, HW42 suggested to approach this problem differently. >>> Instead of integrating the functionality of figuring out the right >>> repositories to reproduce the contents of a buildinfo file into sbuild, >>> write a tool that can drive any package builder (like pbuilder). > > there seems to be a conceptual problem with such an approach. > > For binNMUs, the full changelog entry has to be passed to sbuild or pbuilder. > How does one best pass such a multi-line value via command line options? Would > the best way to pass the changelog entry via the .buildinfo file? And if > pbuilder and sbuild then already are parsing the .buildinfo file, would it not > be better for the debrebuild machinery to be implemented by either in the > first > place? Since this is somewhat relevant to the discussion in the other part of this thread: I don't think this is a conceptual problem. Sure it could be nicer if we don't had binNMUs, but I see no real problem in passing it via cmd line option or via a plain file. I would anyway modify debrebuild to be able to call sbuild/pbuiler/etc. directly and then you are able to use a tempfile cleanly. signature.asc Description: OpenPGP digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Johannes Schauer: > Hi all, > > On Tue, 02 Aug 2016 22:49:00 +0200 Johannes Schauer wrote: >> I was thinking about this issue again and thought that instead of creating a >> wrapper for sbuild which then uses a chroot-setup hook to install the >> dependencies, what I should instead do is to let sbuild itself accept >> .buildinfo files and then do the right thing like: >> >> - use snapshot.d.o to retrieve the right timestamps needed to gather all >>packages >> - mangle the build dependencies such that the source package now depends on >>the exact right package versions and let the resolver figure out the rest >>(thanks Benjamin for that idea) >> - check whether the generated binaries produce the same checksum as given in >>the supplied buildinfo file >> >> But then on IRC, HW42 suggested to approach this problem differently. Instead >> of integrating the functionality of figuring out the right repositories to >> reproduce the contents of a buildinfo file into sbuild, write a tool that can >> drive any package builder (like pbuilder). >> >> I now wrote such a script. > > now that libdpkg-perl comes with support for .buildinfo files, I improved the > script (new version attached) with the following changes: > > - don't use DateTime::Format::Strptime but Time::Piece instead (which is a >perl core module) > - don't use CTRL_INDEX_SRC but CTRL_FILE_BUILDINFO now that dpkg supports >.buildinfo files > - Dpkg::Compression::FileHandle as it is not needed > - the .dsc file name is no longer part of the .buildinfo file, so assemble > the >.dsc file name from the package name and version using > Dpkg::Source::Package > - use the information from the Environment field > - instead of splitting Installed-Build-Depends manually, use >Dpkg::Deps::deps_parse > - instead of using [trusted=yes], retrieve the gpg key of the reproducible >builds repository and verify its fingerprint > - set Binary::apt-get::Acquire::AllowInsecureRepositories to false so that >apt-get fails to update repositories it cannot authenticate > - use Dpkg::Vendor to retrieve the keyring filenames > > Thanks to Guillem Jover for the code review! After discussing this in the irc meeting yesterday I propose that: - we keep it as a separate tool. - put it in a git repo under https://anonscm.debian.org/git/reproducible/ - We have more than enough DDs who are willing to sponsor uploads, so having it in the Debian archive is no problem. - we mainly maintain this as a group. I will try to especially keep an eye on it. Since you have done all the work so far the final decision is obviously up to you. If the above is fine with you I will prepare packaging it during the next week (I also have a few improvements planed). signature.asc Description: OpenPGP digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Johannes Schauer: > Hi, > > On Thu, 10 Nov 2016 05:54:13 -0200 Johannes Schauer wrote: >> On Tue, 02 Aug 2016 22:49:00 +0200 Johannes Schauer wrote: >>> But then on IRC, HW42 suggested to approach this problem differently. >>> Instead of integrating the functionality of figuring out the right >>> repositories to reproduce the contents of a buildinfo file into sbuild, >>> write a tool that can drive any package builder (like pbuilder). > > there seems to be a conceptual problem with such an approach. > > For binNMUs, the full changelog entry has to be passed to sbuild or pbuilder. > How does one best pass such a multi-line value via command line options? What's your problem with passing multi-line value via command line options? > Would the best way to pass the changelog entry via the .buildinfo > file? Not sure about that. If you dislike passing the value via a command line option, just use a plain file? > And if pbuilder and sbuild then already are parsing the .buildinfo > file, would it not be better for the debrebuild machinery to be > implemented by either in the first place? My point for an independent debrebuild was that a) Every builder needs nearly the same functionaly for this. b) It's security relevant since it parses semi-trusted (the .buildinfo) and untrusted (http response from snapshot.d.o) data. So I still think that having this separate of the builder is useful. If sbuild, pbuilder, etc. coordinate this, some kind of library might also be an option. signature.asc Description: OpenPGP digital signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi, On Thu, 10 Nov 2016 05:54:13 -0200 Johannes Schauer wrote: > On Tue, 02 Aug 2016 22:49:00 +0200 Johannes Schauer wrote: > > But then on IRC, HW42 suggested to approach this problem differently. > > Instead of integrating the functionality of figuring out the right > > repositories to reproduce the contents of a buildinfo file into sbuild, > > write a tool that can drive any package builder (like pbuilder). there seems to be a conceptual problem with such an approach. For binNMUs, the full changelog entry has to be passed to sbuild or pbuilder. How does one best pass such a multi-line value via command line options? Would the best way to pass the changelog entry via the .buildinfo file? And if pbuilder and sbuild then already are parsing the .buildinfo file, would it not be better for the debrebuild machinery to be implemented by either in the first place? Thanks! cheers, josch signature.asc Description: signature
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi all,
On Tue, 02 Aug 2016 22:49:00 +0200 Johannes Schauer wrote:
> I was thinking about this issue again and thought that instead of creating a
> wrapper for sbuild which then uses a chroot-setup hook to install the
> dependencies, what I should instead do is to let sbuild itself accept
> .buildinfo files and then do the right thing like:
>
> - use snapshot.d.o to retrieve the right timestamps needed to gather all
>packages
> - mangle the build dependencies such that the source package now depends on
>the exact right package versions and let the resolver figure out the rest
>(thanks Benjamin for that idea)
> - check whether the generated binaries produce the same checksum as given in
>the supplied buildinfo file
>
> But then on IRC, HW42 suggested to approach this problem differently. Instead
> of integrating the functionality of figuring out the right repositories to
> reproduce the contents of a buildinfo file into sbuild, write a tool that can
> drive any package builder (like pbuilder).
>
> I now wrote such a script.
now that libdpkg-perl comes with support for .buildinfo files, I improved the
script (new version attached) with the following changes:
- don't use DateTime::Format::Strptime but Time::Piece instead (which is a
perl core module)
- don't use CTRL_INDEX_SRC but CTRL_FILE_BUILDINFO now that dpkg supports
.buildinfo files
- Dpkg::Compression::FileHandle as it is not needed
- the .dsc file name is no longer part of the .buildinfo file, so assemble the
.dsc file name from the package name and version using Dpkg::Source::Package
- use the information from the Environment field
- instead of splitting Installed-Build-Depends manually, use
Dpkg::Deps::deps_parse
- instead of using [trusted=yes], retrieve the gpg key of the reproducible
builds repository and verify its fingerprint
- set Binary::apt-get::Acquire::AllowInsecureRepositories to false so that
apt-get fails to update repositories it cannot authenticate
- use Dpkg::Vendor to retrieve the keyring filenames
Thanks to Guillem Jover for the code review!
cheers, josch
#!/usr/bin/perl
#
# Copyright 2016 Johannes Schauer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
use strict;
use warnings;
use Dpkg::Control;
use Dpkg::Index;
use Dpkg::Deps;
use Dpkg::Source::Package;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use JSON::PP;
use Time::Piece;
use File::Basename;
eval {
require LWP::Simple;
require LWP::UserAgent;
no warnings;
$LWP::Simple::ua = LWP::UserAgent->new(agent => 'LWP::UserAgent/srebuild');
$LWP::Simple::ua->env_proxy();
};
if ($@) {
if ($@ =~ m/Can\'t locate LWP/) {
die "Unable to run: the libwww-perl package is not installed";
} else {
die "Unable to run: Couldn't load LWP::Simple: $@";
}
}
my $buildinfo = shift @ARGV;
if (not defined($buildinfo)) {
die "need buildinfo filename";
}
# buildinfo support in libdpkg-perl (>= 1.18.11)
my $cdata = Dpkg::Control->new(type => CTRL_FILE_BUILDINFO);
if (not $cdata->load($buildinfo)) {
die "cannot load $buildinfo";
}
my $build_arch = $cdata->{"Build-Architecture"};
if (not defined($build_arch)) {
die "need Build-Architecture field";
}
my $inst_build_deps = $cdata->{"Installed-Build-Depends"};
if (not defined($inst_build_deps)) {
die "need Installed-Build-Depends field";
}
my $srcpkg = Dpkg::Source::Package->new();
$srcpkg->{fields}{'Source'} = $cdata->{"Source"};
$srcpkg->{fields}{'Version'} = $cdata->{"Version"};
my $dsc_fname = $srcpkg->get_basename(1) . ".dsc";
my $environment = $cdata->{"Environment"};
if (not defined($environment)) {
die "need Environment field";
}
$environment =~ s/\n/ /g; # remove newlines
$environment =~ s/^ //; # remove leading whitespace
my $checksums = Dpkg::Checksums->new();
$checksums->add_from_control($cdata);
my @files = $checksums->get_files();
# gather all installed build-depends and figure out the version of base-files
# and dpkg
my $base_files_version;
my $dpkg_version;
my @inst_build_deps = ();
$inst_build_deps = deps_parse($inst_build_deps, reduce_arch => 0, build_dep => 0);
if (! defined $inst_build_deps) {
die "deps_parse failed\n";
}
foreach my $pkg ($inst_build_deps->get_deps()) {
if (! $pkg->isa('Dpkg::Deps::Simple')) {
die "dependency disjunctions are not allowed\n";
}
if (not defined($pkg->{package})) {
die "name undefined";
}
if (defined($pkg->{relation})) {
if ($pkg
Bug#774415: From srebuild sbuild-wrapper to debrebuild
Hi,
On Mon, 09 May 2016 21:07:40 +0200 Johannes Schauer wrote:
> The main disadvantage of the current srebuild implementation is, that it will
> only make use of a single snapshot.d.o timestamp. This makes it impossible to
> reproduce situations where packages are not built in a clean chroot, in a
> partially updated chroot or in a chroot mixing different suites. To assemble
> a chroot with the right package versions, sbuild could retrieve the exact
> right debs from snapshot.d.o.
I was thinking about this issue again and thought that instead of creating a
wrapper for sbuild which then uses a chroot-setup hook to install the
dependencies, what I should instead do is to let sbuild itself accept
.buildinfo files and then do the right thing like:
- use snapshot.d.o to retrieve the right timestamps needed to gather all
packages
- mangle the build dependencies such that the source package now depends on
the exact right package versions and let the resolver figure out the rest
(thanks Benjamin for that idea)
- check whether the generated binaries produce the same checksum as given in
the supplied buildinfo file
But then on IRC, HW42 suggested to approach this problem differently. Instead
of integrating the functionality of figuring out the right repositories to
reproduce the contents of a buildinfo file into sbuild, write a tool that can
drive any package builder (like pbuilder).
I now wrote such a script. It currently supports sbuild or manual installation
by showing the correct sources.list. For both it prints the correct command
line invocations. Advantages over the old sbuild hook-based script attached to
initial post are:
- package versions can come from multiple snapshot timestamps
- theoretically works for more package builders than just sbuild (pbuilder
support is missing because I don't know enough about pbuilder)
- uses Dpkg::Checksums to parse and verify the hash and size fields instead of
doing it manually
- uses apt to download and manage Packages files instead of doing it manually
- allows to add additional repositories like reproducible.alioth.debian.org
(which is hardcoded so far)
- uses base-files and dpkg version to estimate a base Debian release
- drastically reduce number snapshot.d.o API queries by only querying for
missing packages
Limitations:
- There is no nice command line interface with options and switches yet
- You cannot yet supply additional initial archives
(reproducible.alioth.debian.org is hardcoded)
- It only considers Debian main
- It only considers official Debian (and not ports)
- It only considers Debian unstable from snapshot.d.o
- You have to manually run sbuild/pbuilder with the displayed command and then
manually verify if the .buildinfo file stayed the same
What is still needed:
- a good name (I named it debrebuild for now because it is Debian centric and
rebuilds a package that was built before to check if the checksums can be
reproduced locally. This is the main difference to reprotest which does not
require an existing build but checks for reproducibility by building the
same software twice in different environments)
- a nice home for the script to live
- somebody maintaining the software and making it more user friendly by adding
a nice command line interface and writing a README file and/or man page
- maybe let the script execute the sbuild/pbuilder command it suggests to run
as well. This would allow the script to check the output for plausibility.
Usage:
debrebuild.pl package.buildinfo
Depends:
apt-get install --no-install-recommends libdpkg-perl libwww-perl
libdatetime-format-strptime-perl
Have fun!
cheers, josch
#!/usr/bin/perl
#
# Copyright 2016 Johannes Schauer
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
use strict;
use warnings;
use Dpkg::Control;
use Dpkg::Index;
use Dpkg::Compression::FileHandle;
use Dpkg::Deps;
use File::Temp qw(tempdir);
use File::Path qw(make_path);
use JSON::PP;
eval {
require LWP::Simple;
require LWP::UserAgent;
no warnings;
$LWP::Simple::ua = LWP::UserAgent->new(agent => 'LWP::UserAgent/srebuild');
$LWP::Simple::ua->env_proxy();
};
if ($@) {
if ($@ =~ m/Can\'t locate LWP/) {
die "Unable to run: the libwww-perl package is not installed";
} else {
die "Unable to run: Couldn't load LWP::Simple: $@";
}
}
eval {
require DateTime::Format::Strptime;
};
if ($@) {
if ($@ =~ m/Can\'
