Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
Okay. Given the severely constrained manpower from all involved sides, I think the best path forward to make some progress is to continue the review (and eventual merge, hopefully) of my proposed patch in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859867#105. If we want to expand scope, we can do that later. josch, could you take another look at the patch, please? Thank you. On Fri, Jan 5, 2018 at 2:09 PM, Raphael Hertzog wrote: > On Fri, 05 Jan 2018, Michael Stapelberg wrote: > > Raphaël, Benjamin, what’s the current status? It’s been quite a number > > of months since https://bugs.debian.org/859867#122, and I’m still very > > interested in having sbuild easily installable. > > Benjamin never answered on the willingness to host this new infrastructure > within packaging-dev. > > As for me, while I'm still supportive of the idea, I'm afraid that I won't > be able to contribute code in the near future. I have other priorities > related to tracker.debian.org. Hopefully my ideas will help you design > something modular enough to cover the needs of derivatives. Please keep > me in the loop, I might be able to help once it becomes a bit more > concrete and then I might be able to invest some time as part of my work > on Kali. > > Cheers, > -- > Raphaël Hertzog ◈ Debian Developer > > Support Debian LTS: https://www.freexian.com/services/debian-lts.html > Learn to master Debian: https://debian-handbook.info/get/ > -- Best regards, Michael
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
On Fri, 05 Jan 2018, Michael Stapelberg wrote: > Raphaël, Benjamin, what’s the current status? It’s been quite a number > of months since https://bugs.debian.org/859867#122, and I’m still very > interested in having sbuild easily installable. Benjamin never answered on the willingness to host this new infrastructure within packaging-dev. As for me, while I'm still supportive of the idea, I'm afraid that I won't be able to contribute code in the near future. I have other priorities related to tracker.debian.org. Hopefully my ideas will help you design something modular enough to cover the needs of derivatives. Please keep me in the loop, I might be able to help once it becomes a bit more concrete and then I might be able to invest some time as part of my work on Kali. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
Hi, Johannes Schauer writes: > Quoting Simon McVittie (2017-08-20 01:16:12) >> On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote: >> > since sbuild already supports autopkgtest as a backend (and thus also >> > autopkgtest-virt-qemu which vectis seems to be using) I'd be especially >> > interested in hearing which features are missing from sbuild that prevent >> > vectis from making use of that functionality. >> >> The reason vectis doesn't do this is a design choice, not an omission >> in sbuild. > > Thanks for your explanation! I now see how vectis is indeed doing something > that is not in the scope of sbuild itself and that vectis indeed has to be a > wrapper around sbuild and other tools. > > Though maybe it would be a good idea to add parts of your explanations from > your mail to the README.md and package description. It seems at least me and > Raphael were fooled into thinking that vectis' purpose was a different one > than > it actually is. > > Thanks for writing this! So, I take it vectis is out of the picture for the purpose of this bug. Raphaël, Benjamin, what’s the current status? It’s been quite a number of months since https://bugs.debian.org/859867#122, and I’m still very interested in having sbuild easily installable. Thanks! -- Best regards, Michael
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
Quoting Simon McVittie (2017-08-20 01:16:12) > On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote: > > since sbuild already supports autopkgtest as a backend (and thus also > > autopkgtest-virt-qemu which vectis seems to be using) I'd be especially > > interested in hearing which features are missing from sbuild that prevent > > vectis from making use of that functionality. > > The reason vectis doesn't do this is a design choice, not an omission > in sbuild. Thanks for your explanation! I now see how vectis is indeed doing something that is not in the scope of sbuild itself and that vectis indeed has to be a wrapper around sbuild and other tools. Though maybe it would be a good idea to add parts of your explanations from your mail to the README.md and package description. It seems at least me and Raphael were fooled into thinking that vectis' purpose was a different one than it actually is. Thanks for writing this! cheers, josch signature.asc Description: signature
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
On Fri, 11 Aug 2017 at 22:26:20 +0200, Raphael Hertzog wrote: > Simon, you might want to review the history of #859867 and share > your thoughts about this topic and let us know whether vectis > is going to handle all this too. :) Part of the purpose of vectis is to set up and run sbuild in an environment closely resembling production Debian buildds, including running sbuild-createchroot, creating the sbuild user with home directory /nonexistent and so on. Until the production buildds were upgraded to stretch, it even used buildd.debian.org's special branch/fork of jessie sbuild (I'm very glad to have been able to get rid of the hacks I used to make that work). Unlike what is proposed on #859867, all the setup done by vectis is inherently disposable. This is a deliberate design choice. The tarball produced by sbuild-createchroot is cached, but can be regenerated any time; the autopkgtest virtual machine image likewise (although you have to be root and use `vectis bootstrap`, vmdebootstrap or some other manually-created VM to recover if you delete all your cached VM images, since the same VM image is used to create new VM images in `vectis new`); and the actual sbuild installation and setup, since it's sufficiently quick, is done for every build and not cached at all. This is not necessarily the sbuild that every Debian maintainer wants: it's somewhat heavyweight (lots of RAM required for virtualization, lots of setup repeated) and I wouldn't want to build libreoffice in it. However, it's what *I* wanted, and I only maintain small to medium-sized packages anyway. vectis is really just automation of things I was doing already, and things that I would/should have done if they weren't so much work to do manually (so part of its role is helping me to live up to the standards the project expects). The README at https://github.com/smcv/vectis has more details on its design decisions. > On Thu, 10 Aug 2017, Raphael Hertzog wrote: > > We also need > > qemu images for autopkgtest for example. And they also need to be updated > > regularly. `vectis new` creates these images. I haven't yet implemented `vectis refresh` (rebuild e.g. (VM, sbuild tarball, minbase tarball) x (sid, buster, stretch, xenial, artful) x (amd64, i386) in a way that could be done weekly from cron) but I hope to do that soon. > > And as you mentionned, it would be nice to be able to support derivatives > > easily, not only in place of Debian, but next to the usual Debian support. vectis supports Debian and Ubuntu by default, and is reasonably straightforward to configure for other derivatives (in $dayjob I've configured it for SteamOS and the Steam Runtime, and one of my design goals was to make it suitable for other derivatives that Collabora is involved with, like Apertis). On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote: > since sbuild already supports autopkgtest as a backend (and thus also > autopkgtest-virt-qemu which vectis seems to be using) I'd be especially > interested in hearing which features are missing from sbuild that prevent > vectis from making use of that functionality. The reason vectis doesn't do this is a design choice, not an omission in sbuild. I have never seriously tried using the autopkgtest backend, because one of vectis' goals is that if it works in vectis, it should work on the production Debian infrastructure. Real Debian buildds use the schroot backend, therefore vectis must use schroot too - because if it didn't, there would be packages that build successfully in vectis but FTBFS on production infrastructure, because a typical autopkgtest environment has fewer strange quirks than a typical sbuild chroot (e.g. the /nonexistent thing, which has bitten me when working on dbus in the past, and an atypically small package set because no init, bootloader or kernel is needed inside a sbuild chroot). The sbuild/schroot invocation is layered inside a VM, because another vectis design goal is that everything that runs package code is inside a VM. Similarly, vectis knows how to run autopkgtest with the qemu backend (directly) and the lxc and schroot backends (inside a VM), because ci.debian.net currently uses lxc, so vectis should be able to replicate that: it matters for packages like flatpak, bubblewrap and debootstrap, where some tests don't work as desired under lxc and must be skipped or treated as expected failures. I will admit that I don't usually run tests under all three backends before upload - if I was a perfect maintainer, then I would, but it's very slow to run the tests for something like dbus 8 times (build-time, schroot, lxc, qemu, then the same again for i386 because I can't upgrade src:dbus on my amd64 host system until I've also built libdbus-1-3:i386). If the real, production Debian infrastructure ever switches to the autopkgtest backend, so will vectis. I would be happy to add *non-default* modes to vectis where it does things that do not mirror the production infrastructure
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
On Thu, May 18, 2017 at 11:39 PM, Michael Stapelberg wrote: > Sorry for the late reply. > > On Tue, Apr 11, 2017 at 10:39 AM, Johannes Schauer > wrote: > >> Hi, >> >> Quoting Michael Stapelberg (2017-04-08 11:28:12) >> > One area where sbuild sorely lacks is configuration, though: pbuilder >> is very >> > easy to set up, whereas sbuild requires reading through >> > https://wiki.debian.org/sbuild, performing a bunch of steps, only to >> end up >> > with a setup which works fine for unstable, but seems very clumsy when >> > building packages for experimental or backports. >> > >> > One solution to this issue that I can see is to add a new binary >> > package to src:sbuild which — possibly after a brief debconf prompt — >> > performs all the necessary steps to end up with a setup that just >> > works. >> > >> > What are your thoughts on this? Would patches be welcome to add such a >> > package? >> >> patches totally welcome! :D >> > > Cool! I’ll see if I can whip up such a package by working through the wiki > page. > Find attached the first draft of my suggestion. I implemented it as a separate package purely so that I can build it more quickly, but I assume we’d want to fold this into src:sbuild eventually. The resulting package (I built it using dpkg-buildpackage -b) depends on sbuild, schroot, debootstrap, perl-base, lintian. Upon installation, it will create an unstable sbuild schroot, modify its configuration, add all users to sbuild and create a modified ~/.sbuildrc for all users. If you want to read through the entire behavior, I recommend the entrypoints debian/postinst and update-sbuild-chroots. I tested this package on my notebook, which is a Debian installation on which I never had sbuild installed, so I’m reasonably confident that the package works — at least to the point that one gets an sbuild installation that builds packages. I’d be happy about any feedback. Thanks! > > >> >> This is a nice idea! >> >> Maybe these packages could be named sbuild-backend-${foo} where $foo is >> the >> respective backend? At first, a package sbuild-backend-schroot would be >> cool >> because schroot is the default backend. It would be nice if that would >> set up >> sbuild schroots for stable-backports, unstable and experimental. Maybe >> that >> package should also install and activate cron-jobs to regularly update >> those >> schroots? >> > > Which other backends even are there, and do we really need to offer our > users that choice, when we’re talking about a package with sane defaults > for people who prefer not to make these choices right now? :) > > >> >> What irks me is, that this setup would be Debian specific. It doesn't >> make much >> sense for Debian's downstreams to have have schroots for Debian. Maybe the >> distribution name should be part of the package name? Or maybe it should >> be >> easy for downstreams that care to override the set of distributions the >> schroots are created for? >> > > Putting Debian into the package name seems reasonable. > > How about sbuild-debian-dev-setup? > > I originally thought of sbuild-debian-setup, but it could be confused to > mean “a setup which reflects Debian’s official setup”, i.e. the buildds. > > The “setup” suffix to me conveys that this package offers only > configuration, not new software. Perhaps there’s a more idiomatic term for > that in Debian? > > >> >> Thanks! >> >> cheers, josch >> > > > > -- > Best regards, > Michael > -- Best regards, Michael sbuild-debian-setup.tar.bz2 Description: BZip2 compressed data
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
Sorry for the late reply. On Tue, Apr 11, 2017 at 10:39 AM, Johannes Schauer wrote: > Hi, > > Quoting Michael Stapelberg (2017-04-08 11:28:12) > > One area where sbuild sorely lacks is configuration, though: pbuilder is > very > > easy to set up, whereas sbuild requires reading through > > https://wiki.debian.org/sbuild, performing a bunch of steps, only to > end up > > with a setup which works fine for unstable, but seems very clumsy when > > building packages for experimental or backports. > > > > One solution to this issue that I can see is to add a new binary > > package to src:sbuild which — possibly after a brief debconf prompt — > > performs all the necessary steps to end up with a setup that just > > works. > > > > What are your thoughts on this? Would patches be welcome to add such a > > package? > > patches totally welcome! :D > Cool! I’ll see if I can whip up such a package by working through the wiki page. > > This is a nice idea! > > Maybe these packages could be named sbuild-backend-${foo} where $foo is the > respective backend? At first, a package sbuild-backend-schroot would be > cool > because schroot is the default backend. It would be nice if that would set > up > sbuild schroots for stable-backports, unstable and experimental. Maybe that > package should also install and activate cron-jobs to regularly update > those > schroots? > Which other backends even are there, and do we really need to offer our users that choice, when we’re talking about a package with sane defaults for people who prefer not to make these choices right now? :) > > What irks me is, that this setup would be Debian specific. It doesn't make > much > sense for Debian's downstreams to have have schroots for Debian. Maybe the > distribution name should be part of the package name? Or maybe it should be > easy for downstreams that care to override the set of distributions the > schroots are created for? > Putting Debian into the package name seems reasonable. How about sbuild-debian-dev-setup? I originally thought of sbuild-debian-setup, but it could be confused to mean “a setup which reflects Debian’s official setup”, i.e. the buildds. The “setup” suffix to me conveys that this package offers only configuration, not new software. Perhaps there’s a more idiomatic term for that in Debian? > > Thanks! > > cheers, josch > -- Best regards, Michael
Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging
Hi, Quoting Michael Stapelberg (2017-04-08 11:28:12) > One area where sbuild sorely lacks is configuration, though: pbuilder is very > easy to set up, whereas sbuild requires reading through > https://wiki.debian.org/sbuild, performing a bunch of steps, only to end up > with a setup which works fine for unstable, but seems very clumsy when > building packages for experimental or backports. > > One solution to this issue that I can see is to add a new binary > package to src:sbuild which — possibly after a brief debconf prompt — > performs all the necessary steps to end up with a setup that just > works. > > What are your thoughts on this? Would patches be welcome to add such a > package? patches totally welcome! :D This is a nice idea! Maybe these packages could be named sbuild-backend-${foo} where $foo is the respective backend? At first, a package sbuild-backend-schroot would be cool because schroot is the default backend. It would be nice if that would set up sbuild schroots for stable-backports, unstable and experimental. Maybe that package should also install and activate cron-jobs to regularly update those schroots? What irks me is, that this setup would be Debian specific. It doesn't make much sense for Debian's downstreams to have have schroots for Debian. Maybe the distribution name should be part of the package name? Or maybe it should be easy for downstreams that care to override the set of distributions the schroots are created for? Thanks! cheers, josch signature.asc Description: signature