schroot default chroot path (was: Re: Intent to commit craziness - source package unpacking)
Hi, Quoting Ian Jackson (2016-11-30 12:11:22) > ~Johannes Schauer writes ("Re: Intent to commit craziness - source package > unpacking"): > > sbuild supports multiple backends. The default backend is schroot. How to > > get > > from "apt-get install sbuild" to actually building packages depends on the > > backend used. For the default schroot backend, sbuild provides the > > sbuild-createchroot tool which runs debootstrap and creates a schroot config > > file. So if you stay with the defaults, then you run: > > > > $ apt-get install sbuild > > $ sbuild-createchroot unstable /srv/chroot/unstable-amd64 > > WIBNI the final argument had a default ? :-) it would. If anybody knows a good argument for a default location for schroot containers then I can add such a default. I am not aware of such a standard path for system-wide schroot directories or tarballs. Anybody with a good argument for such a default location can feel free to open a wishlist bug against sbuild and then we can discuss this there. Probably good to do this together with the schroot maintainer (Roger Leigh in CC). Thanks! cheers, josch signature.asc Description: signature ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
~Johannes Schauer writes ("Re: Intent to commit craziness - source package unpacking"): > sbuild supports multiple backends. The default backend is schroot. How to get > from "apt-get install sbuild" to actually building packages depends on the > backend used. For the default schroot backend, sbuild provides the > sbuild-createchroot tool which runs debootstrap and creates a schroot config > file. So if you stay with the defaults, then you run: > > $ apt-get install sbuild > $ sbuild-createchroot unstable /srv/chroot/unstable-amd64 WIBNI the final argument had a default ? :-) Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Hi, Quoting Ian Jackson (2016-10-04 18:55:14) > Russ Allbery writes ("Re: Intent to commit craziness - source package > unpacking"): > > If sbuild is now at the point where you can just apt-get install sbuild, > > run a single setup command, and be ready to build packages (which is where > > cowbuilder is right now), I personally would be happy to use something > > that's a bit closer to what the buildds are doing. > > There is sbuild-setupchroot or something. I do find that I don't want > it because I like fiddling with the config, so I just use lower level > commands myself. to answer above questions: sbuild supports multiple backends. The default backend is schroot. How to get from "apt-get install sbuild" to actually building packages depends on the backend used. For the default schroot backend, sbuild provides the sbuild-createchroot tool which runs debootstrap and creates a schroot config file. So if you stay with the defaults, then you run: $ apt-get install sbuild $ sbuild-createchroot unstable /srv/chroot/unstable-amd64 $ sbuild mypackage.dsc The sbuild version in stable also requires you to generate a gpg keypair (instructions are at https://wiki.debian.org/sbuild) but this inconvenience was dropped when squeeze (which had a too-old apt) reached end of life this year. If you use another backend than schroot (like lxc, qemu or ssh) you are so far on your own with preparing the build chroot. Please report any wishlist bugs that would make your life with sbuild easier in the Debian bts. Thanks! cheers, josch signature.asc Description: signature ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Russ Allbery writes ("Re: Intent to commit craziness - source package unpacking"): > Ian Jackson <ijack...@chiark.greenend.org.uk> writes: > > I'm not sure of the logic behind that. I don't think dgit helps much > > with the kind of tasks that pristine-tar helps with. > > The main benefit of pristine-tar is that you can clone the development > repository on any random host and do package development, build local > packages for testing, and do a package upload without having to locate any > additional pieces. I haven't been following dgit development closely, but > it did sound like you were addressing the same use case. dgit will fetch the orig tarball for you. So you don't have to "locate" the pieces, but you do have to pay the download cost. > If sbuild is now at the point where you can just apt-get install sbuild, > run a single setup command, and be ready to build packages (which is where > cowbuilder is right now), I personally would be happy to use something > that's a bit closer to what the buildds are doing. There is sbuild-setupchroot or something. I do find that I don't want it because I like fiddling with the config, so I just use lower level commands myself. > However, cowbuilder does just work, in my experience, and it's nice to not > have to change. Of course. I'm sorry I stepped into the argument about which chroot build tool is best :-). Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package unpacking"): > On Mon, Oct 03, 2016 at 04:15:08PM +0100, Ian Jackson wrote: > [..snip..] > > Recommends: pristine-tar (>= 0.5) ...> > > > pristine-tar has been declared unmaintainable by its original > > author and abandoned. ... > > Certainly dgit users do not need pristine-tar. But our dependency > > system does not allow us to honour only direct Recommends and not > > transitive ones. > > Looking at git.debian.org I found plenty of users. I did an archive > import of sid during Debconf and was only ran into 20 pristine-tar > failures (bugs yet to be filed). Interesting. > From the discussions at DC16 we're on our way to make this a hard > dependency: > > http://lists.sigxcpu.org/pipermail/git-buildpackage/2016-July/000143.html > > The only thing I can think of (since we will keep support for not using > pristine-tar nevertheless) is using: > > Recommends: pristine-tar | dgit I'm not sure of the logic behind that. I don't think dgit helps much with the kind of tasks that pristine-tar helps with. > > Recommends: cowbuilder<= jessie > > Recommends: cowbuilder | pbuilder | sbuild<= sid ... > gbp buildpackage has integration with pbuilder/cowbuilder (via > git-builder) and I know people are using it since its better integrated > into gbp since you don't need additional and it's documented in the > manual. The sbuild dependency is there to have people not pull in > cowbuilder/pbuilder so they can use --git-builder=sbuild. Ah. > Not sure what can be done here. It sounds like it should be left as-is, TBH. > > Depends: devscripts > > > > devscripts is very full of commands with poor namespacing. It > > also has an enormous dependency chain. > > > > Unfortunately dgit has a dependency on devscripts too. Maybe we > > should work to take the pieces of devscripts that we really need > > and put them in something else, or something. > > We're mostly using dch with "gbp dch" and I would also be happy to have > the dependency chain shortened. If it were my package, and that was all I depended on devscripts for, I would drop it entirely. I think it's fair to expect someone who uses `gbp dch' to install the package containing dch. But this is a matter of taste. dgit has a much harder dependency because dgit push uses dput. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Hi Ian, On Mon, Oct 03, 2016 at 04:15:08PM +0100, Ian Jackson wrote: [..snip..] > Because I'm a pernickety type of person I reviewed the dependencies of > git-buildpackage. I have some qualms about the following > dependencies: > > Recommends: pristine-tar (>= 0.5) > > pristine-tar has been declared unmaintainable by its original > author and abandoned. > > IDK know what proportion of actual git trees that gbp users will > encounter would break without pristine-tar. Perhaps this > dependency can be demoted to Suggests, but I don't really know. > > Certainly dgit users do not need pristine-tar. But our dependency > system does not allow us to honour only direct Recommends and not > transitive ones. Looking at git.debian.org I found plenty of users. I did an archive import of sid during Debconf and was only ran into 20 pristine-tar failures (bugs yet to be filed). >From the discussions at DC16 we're on our way to make this a hard dependency: http://lists.sigxcpu.org/pipermail/git-buildpackage/2016-July/000143.html The only thing I can think of (since we will keep support for not using pristine-tar nevertheless) is using: Recommends: pristine-tar | dgit > Recommends: cowbuilder<= jessie > Recommends: cowbuilder | pbuilder | sbuild<= sid > > Many users of dgit will never want to build a package for upload. > This is probably true of gbp users too. I'm not sure why it's > valuable to have this as a Recommends dependency for gbp. > > I think more people now are using sbuild. I'm not sure that > pulling in cowbuilder on systems where the user has not yet > installed such a tool is right. gbp buildpackage has integration with pbuilder/cowbuilder (via git-builder) and I know people are using it since its better integrated into gbp since you don't need additional and it's documented in the manual. The sbuild dependency is there to have people not pull in cowbuilder/pbuilder so they can use --git-builder=sbuild. Not sure what can be done here. > Depends: devscripts > > devscripts is very full of commands with poor namespacing. It > also has an enormous dependency chain. > > Unfortunately dgit has a dependency on devscripts too. Maybe we > should work to take the pieces of devscripts that we really need > and put them in something else, or something. We're mostly using dch with "gbp dch" and I would also be happy to have the dependency chain shortened. > Overall I don't think these are an impediment. But since I had done > the review, I thought I'd share my thoughts. Cheers, -- Guido ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Guido Günther writes ("Re: Intent to commit craziness - source package > unpacking"): > > 'gbp pq import' does have 'debian/patches' since it just puts the > > patches that are in debian/patches on top of the unpatched source > > tree. In contrast to your solution it doesn't try to be able to > > roundtrip without changes for any given series on "gbp pq export". This > > is only true if the series was also created by "gbp pq" (or adheres to > > what git-format-patch does). > > Currently the output of dpkg-source --commit doesn't look much like > the output of git-format-patch. > > I have tried to make dgit produce patches (when it needs to produce > patches) that look like dpkg-source --commit. But maybe it should > produce patches using git-format-patch (or that look like > git-format-patch). I have looked at the specs (including DEP-3) and the behaviour of the various tools a bit more. I have decided that when I should replace dgit's adhoc algorithm for generating a DEP-3 compliant commit message, with a call to git-format-patch. (The actual patch body has to come from dpkg-soure, because of the special handling of debian/, etc. This code path is used when a dgit user is pushing and needs to convert raw git commits into `3.0 (quilt)' patches.) And, after having half-written a DEP-3 importer (ie a thing that turns a DEP-3 patch message into a git commit), and investigating the behaviour of various tools, I have decided I should just use gbp pq. (This will replace the repeated use of dpkg-source --before-build as a described in my previous mail.) This will mean that dgit will grow a hard dependency on git-buildpackage. Because I'm a pernickety type of person I reviewed the dependencies of git-buildpackage. I have some qualms about the following dependencies: Recommends: pristine-tar (>= 0.5) pristine-tar has been declared unmaintainable by its original author and abandoned. IDK know what proportion of actual git trees that gbp users will encounter would break without pristine-tar. Perhaps this dependency can be demoted to Suggests, but I don't really know. Certainly dgit users do not need pristine-tar. But our dependency system does not allow us to honour only direct Recommends and not transitive ones. Recommends: cowbuilder<= jessie Recommends: cowbuilder | pbuilder | sbuild<= sid Many users of dgit will never want to build a package for upload. This is probably true of gbp users too. I'm not sure why it's valuable to have this as a Recommends dependency for gbp. I think more people now are using sbuild. I'm not sure that pulling in cowbuilder on systems where the user has not yet installed such a tool is right. Depends: devscripts devscripts is very full of commands with poor namespacing. It also has an enormous dependency chain. Unfortunately dgit has a dependency on devscripts too. Maybe we should work to take the pieces of devscripts that we really need and put them in something else, or something. Overall I don't think these are an impediment. But since I had done the review, I thought I'd share my thoughts. Regards, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package unpacking"): > The authorship and subject information is taken from the patch if > available or made up from the filename and committer when not. > > If patches in the series file are in subdirs of debian/patches we store > that in "Gbp-Pq: topic" in the commit message. The patch name itself > is stored in "Gbp-Pq: Name" we can reproduce it on "gbp pq export" > independent from the patch's subject. Right. > > Bear in mind that because the output of gbp-pq import doesn't > > contain debian/patches, I would need to rewrite its output (perhaps > > with git-filter-branch). > > 'gbp pq import' does have 'debian/patches' since it just puts the > patches that are in debian/patches on top of the unpatched source > tree. In contrast to your solution it doesn't try to be able to > roundtrip without changes for any given series on "gbp pq export". This > is only true if the series was also created by "gbp pq" (or adheres to > what git-format-patch does). Currently the output of dpkg-source --commit doesn't look much like the output of git-format-patch. I have tried to make dgit produce patches (when it needs to produce patches) that look like dpkg-source --commit. But maybe it should produce patches using git-format-patch (or that look like git-format-patch). Thanks for suggesting libvirt as an example. I will play about with it. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Guido Günther writes ("Re: Intent to commit craziness - source package > unpacking"): > > Hi Ian, > > Hi, thanks for your attention. Please disregard this email. I hadn't finished editing it! Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Hi Ian, On Wed, Sep 28, 2016 at 11:09:03AM +0100, Ian Jackson wrote: > Ian Jackson writes ("Re: Intent to commit craziness - source package > unpacking"): > > Thanks for your comments. I feel unblocked :-). > > So, I now intend to go and implement my plan. > > There will be a little while (perhaps a few weeks) before I am in a > postion to release this in dgit 2.0. But after I do that, I will not > want to change the import algorithm again: it is important that the > imports be as stable as possible. > > So now would be a good time for maintainers of git packaging tools (eg > git-dpm and gpb) to have an opinion about the detail of the generated > pseudohistory - in particular, the detail of the commits generated > from the `3.0 (quilt)' dpkg-source patches. >From what I understand the format to produce is not compatible in what "gbp pq" currently expects, that is just one commit per patch without any chnages to other files in debian/ (like the series file). 'gbp pq' is just a helper for handling the quilt patches, not more. I don't understand yet how you expect the actual workflow between gbp and dgit to look like but I would be happy to have a look at a prototype dgit created history. I can e.g. imagine telling "gbp pq" to filter out chnages in debian/ during patch creation. > Also, I would welcome suggestions for what kind of compatibility test > I could perform on such a series of commits. dgit has an extensive > test suite (advertised via autopkgtest) which would be well-suited to > such a compatibility test. > > An example of such a tree might be, "split out the patch queue part of > the git pseudohistory and feed it to gbp-pq, asking gbp-pq to > regenerate the source package, and expect the output to be identical > to the original input source package". Guido, if I get the > preconditions right, should I expect such a test to pass ? Is there a > risk it would break in the future due to changes in gbp-pq's > conversion algorithm ? It would be nice to have "gbp pq" reproduce debian/patches identically on such a tree but I would rather have a look at how the dgit created history finally looks once you implemented it. gbp-pq's conversion algorithm is not expected to change (at least in the default configuration, I have some other ideas about patch handling but that would not modify the current workflow ). Hope that helps, -- Guido > I confess that I am less familiar with git-dpm. I don't know what I > should be thinking about to try to make the output most useful to > git-dpm users. > > (I also don't know whether the goals of helping git-dpm users and > gbp-pq users, and potentially users of any other tools, are in > conflict. > > It would be annoying if these tools would disagree about the best form > of import of a particular patch queue: the import algorithm should be > the same for different dgit users, so I wouldn't be able to make this > a per-user configuration option and would have to choose..) > > Thanks, > Ian. > > -- > Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. > > If I emailed you from an address @fyvzl.net or @evade.org.uk, that is > a private address which bypasses my fierce spamfilter. > > ___ > vcs-pkg-discuss mailing list > vcs-pkg-discuss@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss > ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Thanks for your comments. I feel unblocked :-). So, I now intend to go and implement my plan. There will be a little while (perhaps a few weeks) before I am in a postion to release this in dgit 2.0. But after I do that, I will not want to change the import algorithm again: it is important that the imports be as stable as possible. So now would be a good time for maintainers of git packaging tools (eg git-dpm and gpb) to have an opinion about the detail of the generated pseudohistory - in particular, the detail of the commits generated from the `3.0 (quilt)' dpkg-source patches. Also, I would welcome suggestions for what kind of compatibility test I could perform on such a series of commits. dgit has an extensive test suite (advertised via autopkgtest) which would be well-suited to such a compatibility test. An example of such a tree might be, "split out the patch queue part of the git pseudohistory and feed it to gbp-pq, asking gbp-pq to regenerate the source package, and expect the output to be identical to the original input source package". Guido, if I get the preconditions right, should I expect such a test to pass ? Is there a risk it would break in the future due to changes in gbp-pq's conversion algorithm ? I confess that I am less familiar with git-dpm. I don't know what I should be thinking about to try to make the output most useful to git-dpm users. (I also don't know whether the goals of helping git-dpm users and gbp-pq users, and potentially users of any other tools, are in conflict. It would be annoying if these tools would disagree about the best form of import of a particular patch queue: the import algorithm should be the same for different dgit users, so I wouldn't be able to make this a per-user configuration option and would have to choose..) Thanks, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Intent to commit craziness - source package unpacking
tl;dr: * dpkg developers, please tell me whether I am making assumptions that are likely to become false. Particularly, on the behaviour of successive runs of dpkg-source --before-build with successively longer series files. * git-buildpackage and git-dpm developers, please point me to information about what metadata to put into the commit message for a git commit which represents a dpkg-source quilt patch. I would like these commits to be as convenient for gbp and git-dpm users as possible. Hi. Currently when dgit needs to import a .dsc into git, it just uses dpkg-source -x, and git-add. The result is a single commit where the package springs into existence fully formed. This is not as good as it could be. I would like to represent (in the git pseudohistory) the way that the resulting tree is constructed from the input objects. In particular, I would like to: represent the input tarballs as a commit each (which all get merged together as if by git merge -s subtree), and for quilt packages, each patch as a commit. But I want to avoid (as much as possible) reimplementing the package extraction algorithm in dpkg-source. dpkg-source does not currently provide interfaces that look like they are intended for what I want to do. And dgit wants to work with old versions of dpkg, so I don't want to block on getting such interfaces added (even supposing that a sane interface could be designed, which is doubtful). So I intend to do as follows. (Please hold your nose.) * dgit will untar each input tarball (other than the Debian tarball). This will be done by scanning the .dsc for things whose names look like (compressed) tarballs, and using the interfaces provided by Dpkg::Compression to get at the tarball. Each input tarball unpack will be done separately, and will be followed by git-add and git-write tree, to obtain a git tree object corresponding to the tarball contents. That tree object will be made into a commit object with no parents. (The package changelog will be searched for the earliest version with the right upstream version component, and the information found there used for the commit object's metadata.) * dgit will then run dpkg-source -x --skip-patches. Again, git plumbing will be used to make this into a tree and a commit. The commit will have as parents all the tarballs previous mentioned. The metadata will come from the .dsc and/or the final changelog entry. * dgit will look to see if the package is `3.0 (quilt)' and if so whether it has a series file. (dgit already rejects packages with distro-specific series files, so we need worry only about a single debian/patches/series file.) If there is a series file, dgit will read it into memory. It will then iterate over the series file, and each time: - write into its playground a series file containing one more non-comment non-empty line to previously - run dpkg-source --before-build (which will apply that additional patch) - make git tree and commit objects, using the metadata from the relevant patch file to make the commit (if available) - each commit object has as a parent the previous commit (either the previous commit, or the commit resulting from dpkg-source -x) After this the series file has been completely rewritten. * dgit will then run one final invocation of dpkg-source --before-build. This ought not to produce any changes, but if it does, they will be represented as another commit. * As currently, there will be a final no-change-to-the-tree pseudomerge commit which stitches the package into the relevant dgit suite branch; ie something that looks as if it was made with git merge -s ours. * As currently, dgit will take steps so that none of the git trees discussed above contain a .pc directory. This has the following properties: * Each input tarball is represented by a different commit; in usual cases these commits will be the same for every upload of the same upstream version. * For `3.0 (quilt)' each patch's changes to the upstream files appears as a single git commit (as is the effect of the debian tarball). For `1.0' non-native, the effect of the diff is represented as a commit. So eg `git blame' will show synthetic commits corresponding to the correct parts of the input source package. * It is possible to `git-cherry-pick' etc. commits representing `3.0 (quilt)' patches. It is even possible fish out the patch stack as git branch and rebase it elsewhere etc., since the patch stack is represented as a contiguous series of commits which make only the relevant upstream changes. * Every orig tarball in the source package is decompressed twice, but disk space for only one extra copy of its unpacked contents is needed. (The converse would be possible in principle but would be very hard to arrange with the current interfaces provided by the various tools.) *