Re: Simpler git workflow for packaging with upstreamless repositories
Hello, On Sat 07 Dec 2024 at 05:50pm GMT, Holger Levsen wrote: > On Sat, Dec 07, 2024 at 10:39:43PM +0500, Andrey Rakhmatullin wrote: >> No, there is clearly no consensus on unifying any workflows. Everyone >> thinks their workflow is superior and canneeds to stay. > > I agree with the first sentence but I think the 2nd sentence is too much > drama. > > Those many workflows exists for reasons, some good, some not so good. Right. And there are many package-specific needs. -- Sean Whitton
Re: Simpler git workflow for packaging with upstreamless repositories
On Sat Dec 7, 2024 at 5:39 PM GMT, Andrey Rakhmatullin wrote: No, there is clearly no consensus on unifying any workflows. Everyone thinks their workflow is superior and canneeds to stay. I'm more optimistic than this. I don't think we'll ever reach *one* workflow, but I think there may be a steadily emerging consensus about a common "highway" workflow. So long as it's not mandatory, and we recognise that many packages will not be suited to it, I hope even those who won't be transitioning to it would recognise the value of having it. On Sat, Dec 07, 2024 at 02:34:20PM +0100, Andrea Pappacoda wrote: Couldn't we decide to unify on a single "front end" command, which chooses different backends automatically depending on the situation? I can see the appeal of having a single "front-end" for this (and many other actions), especially for newcomers, but I think the general approach of introducing a new front-end wrapper would cause more harm than good. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Simpler git workflow for packaging with upstreamless repositories
On Sat, Dec 07, 2024 at 10:39:43PM +0500, Andrey Rakhmatullin wrote: > No, there is clearly no consensus on unifying any workflows. Everyone > thinks their workflow is superior and canneeds to stay. I agree with the first sentence but I think the 2nd sentence is too much drama. Those many workflows exists for reasons, some good, some not so good. Even if we agreed on the one superiour workflow tomorrow (which won't happen just because some people think that would be a good idea, but let's assume so), it would still take years to achieve. Which is not to say that cannot be done, but changing 30k source packages takes years, probably rather a decade. Changing 2000 people decades old habbits will probably take even longer. What could be done much more easily however, would be to agree on one default recommended workflow & toolchain for NEW packages and people. Maybe. OTOH, some of us do this already, eg the rust team. And because we are the Debian rust team, we also have exceptions to this rule... ;) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Klimakatastrophe bedeutet kompletter sozialer Kollaps. signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On 07/12/2024 14:34, Andrea Pappacoda wrote: It seems that we're all agreeing on trying to unify our different workflows as much as possible. Why do we have `git deborig`, `gbp export-orig`, `origtargz`, etc.? Couldn't we decide to unify on a single "front end" command, which chooses different backends automatically depending on the situation? It would be a starting point. To new contributors, we could say, for example, "to generate the tarball, just run origtargz", independently of whether they use gbp, git-debrebase, no git at all, etc. Bye! https://xkcd.com/927/ --alec
Re: Simpler git workflow for packaging with upstreamless repositories
On Sat, Dec 07, 2024 at 02:34:20PM +0100, Andrea Pappacoda wrote: > Hi all, I've tried catching up with the whole thread, but didn't fully yet. > So excuse me if this has been asked/answered before. > > On Tue Dec 3, 2024 at 3:40 AM CET, Sean Whitton wrote: > > Then just make one: 'git deborig'. > > > > I appreciate not everyone is happy with this, but it solves the problem. > > It seems that we're all agreeing on trying to unify our different workflows > as much as possible. No, there is clearly no consensus on unifying any workflows. Everyone thinks their workflow is superior and canneeds to stay. > Why do we have `git deborig`, `gbp export-orig`, `origtargz`, etc.? They are parts of different incompatible workflows. > Couldn't we decide to unify on a single "front end" command, which > chooses different backends automatically depending on the situation? That seems unlikely. It can't be a command that runs the full workflow and it can't be a set of separate commands that run separate parts of different workflows because the parts themselves are unlikely to be possible to uinify. > It would be a starting point. To new contributors, we could say, for > example, "to generate the tarball, just run origtargz", independently of > whether they use gbp, git-debrebase, no git at all, etc. Ideally one shouldn't need to run any separate commands to generate the tarball from a repo. E.g. the gbp workflow doesn't require this. Additionally, e.g. gbp expects the tarball to be in the build-dir, while no unrelated tools could know the location of the gbp build-dir. See? -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi all, I've tried catching up with the whole thread, but didn't fully yet. So excuse me if this has been asked/answered before. On Tue Dec 3, 2024 at 3:40 AM CET, Sean Whitton wrote: Then just make one: 'git deborig'. I appreciate not everyone is happy with this, but it solves the problem. It seems that we're all agreeing on trying to unify our different workflows as much as possible. Why do we have `git deborig`, `gbp export-orig`, `origtargz`, etc.? Couldn't we decide to unify on a single "front end" command, which chooses different backends automatically depending on the situation? It would be a starting point. To new contributors, we could say, for example, "to generate the tarball, just run origtargz", independently of whether they use gbp, git-debrebase, no git at all, etc. Bye!
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Dec 03, 2024 at 10:40:03AM +0800, Sean Whitton wrote: > >> > One possible rebuttal to this is "gbp needs to do the right thing then". > >> > Currently gbp by default generates a broken tarball, which is also a > >> > source of confusion for many. > >> > >> Do you have a bug report number? > > > > No. > > I've found #902249 which is titled "Pull tarballs from the archive (or > > upstream location)", maybe that's the one you think about. I haven't read > > it, except for the "I hoped we could stay out of that business in 2018 but > > since tarballs are still _the_ _thing_ in Debian" line, which I liked. > > > > For the avoidance of doubt, I don't think gbp *can* do the right thing > > there. It's not my rebuttal. Maybe gbp should just refuse to generate a > > random tarball from a commit-ish and let^Wrequire people to provide one or > > provide a way to generate one in a correct way. > > 'origtargz' from devscripts usually does the right thing. Yeah, but it's a separate command. Some parts of this discussion are specifically about the straightforward gbp-clone+gbp-buildpackage workflow. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hello, On Thu 28 Nov 2024 at 10:29am -10, Theodore Ts'o wrote: > *) As much as possible, we want to be able to use the unmodified >source files are officially released by upstream. Which might be a >tarball and/or a signed git tag. I ignore this completely, and I'm not the only one. Even for traditionally maintained software like Emacs, Rob and I push upstream-signed git tags to dgit.debian.org instead, and use 'git archive' to generate a tarball to satisfy dak. -- Sean Whitton signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hello, On Tue 26 Nov 2024 at 11:20pm +05, Andrey Rakhmatullin wrote: > The archive, when the tarball is already there. > > These suggestions never discuss what to do when the tarball was never > uploaded yet, even I didn't discuss that for simplicity. Then just make one: 'git deborig'. I appreciate not everyone is happy with this, but it solves the problem. -- Sean Whitton
Re: Simpler git workflow for packaging with upstreamless repositories
Hello, On Tue 26 Nov 2024 at 11:28pm +05, Andrey Rakhmatullin wrote: > On Tue, Nov 26, 2024 at 06:53:01PM +0100, Mechtilde Stehmann wrote: >> > One possible rebuttal to this is "gbp needs to do the right thing then". >> > Currently gbp by default generates a broken tarball, which is also a >> > source of confusion for many. >> >> Do you have a bug report number? > > No. > I've found #902249 which is titled "Pull tarballs from the archive (or > upstream location)", maybe that's the one you think about. I haven't read > it, except for the "I hoped we could stay out of that business in 2018 but > since tarballs are still _the_ _thing_ in Debian" line, which I liked. > > For the avoidance of doubt, I don't think gbp *can* do the right thing > there. It's not my rebuttal. Maybe gbp should just refuse to generate a > random tarball from a commit-ish and let^Wrequire people to provide one or > provide a way to generate one in a correct way. 'origtargz' from devscripts usually does the right thing. -- Sean Whitton
Re: Simpler git workflow for packaging with upstreamless repositories
On Wed Nov 27, 2024 at 4:30 AM GMT, Otto Kekäläinen wrote: > How common debian/gbp.conf points at something else: perhaps gbp's > defaults are not good, if that many packages need to override them. First of all may I ask you to not use terms like 'not good' as it may come off negative towards the maintainer of that software. Guido has done an amazing job with git-buildpackage and we certainly want him to continue iterating on it. I was really surprised to receive this feedback. I agree with using intentional language and not denigrating other's work. I spent a few days reflecting on what I wrote and how you've responded to it, and I haven't come to a conclusion as to whether I agree with you or not. On the one hand, juxtaposing "not good" with "gbp" could be taken badly, and using something like "not correct" or "not appropriate" may have avoided that; on the other, I believe I was clear in expressing that I was talking about the software's defaults rather than the software itself. I also suggest you to participate in gbp development, as currently it is 95% Guido alone. At https://salsa.debian.org/agx/git-buildpackage/-/commits/master you can for example suggest changes to those defaults along with a migration path, or you can help with just improving the documentation so people more easily end up choosing the right options. Presently I prefer not to use gbp, so I don't think this would be an efficient use of my time. I'm also undecided on whether gbp should be recommended as the default tool that we recommend to developers for managing packages. I feel that engaging with your efforts on DEP18, considering how to progress DEP14 and the related discussions on -devel are a better use of my resource at the present moment. Secondly, it is perfectly valid for evey single package to have a debian/gbp.conf and I would in fact prefer that. For every upstream we need to have metadata on: - do they have tarball releases (pristine-tar true/false) Please don't infer the answer to "does upstream have tarball releases" from "pristine-tar true/false", as they are not the same thing. I'm sure there are packages repositories where upstream's tarball is ignored, a "git archive"-produced tarball, or a GitHub release-derived tarball, is used instead, *and* the pristine-tar branch duly populated (which again is independent from whether the maintainers have provided a debian/gbp.conf with a pristine-tar key value in it.) Likewise there are package repositories where the packaging is based on an upstream tarball, but pristine-tar is not used, and/or its use not documented in a debian/gbp.conf. More generally I think the *right* place for much of this metadata is not gbp.conf but should be somewhere more tool-agnostic, at least unless we actually agree to elevate gbp to the status of "use this unless you have a very good reason not to". -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Simpler git workflow for packaging with upstreamless repositories
Le 2024-11-29 18:35, Jonas Smedegaard a écrit : Quoting sre4e...@free.fr (2024-11-29 16:57:23) opened 15 years and a few days ago. Great that you took "filing a bugreport" to imply checking first if the issue was already reported. Well, the point I wanted to make here was more about the speculative far better traction of reporting bugs ... I would even dare to write that the continued existence of this bug report alone for more than 15 years, with all its comments, proves perfectly that you were wrong about that. And I do really think that this is really unfortunate, but that's still a sad and hard fact. Which doesn't imply that asking for the same thing on a mailing-list is any better in terms of getting things done. Cheers, -- Julien Plissonneau Duquène
Re: Simpler git workflow for packaging with upstreamless repositories
Quoting sre4e...@free.fr (2024-11-29 16:57:23) > Le 2024-11-29 15:12, Jonas Smedegaard a écrit : > > Quoting sre4e...@free.fr (2024-11-29 13:29:05) > >> > >> Would it be possible to make it the default when there is already an > >> existing `pristine-tar` branch in the repo being worked on? > > > > Please consider filing that as a bugreport against bit-buildpackage - > > I suspect that has a far better traction than asking speculatively in > > this mailinglist and assuming that others picks up your thought and > > does > > that task for you. > > That would be a duplicate though, someone else already had that great > idea: see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=557606 > opened 15 years and a few days ago. Great that you took "filing a bugreport" to imply checking first if the issue was already reported. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private
Re: Simpler git workflow for packaging with upstreamless repositories
Le 2024-11-29 10:58, Simon Josefsson a écrit : The only record of this is indirectly with the maintainer signing the *.changes file during package upload. But that is weak (only successfully uploaded packages are protected, not work-in-progress) and not widely audited (*.changes files aren't stored forever, or are they?). I would like to know that as well, and if there is an automatable way to retrieve them even long after they are processed. Cheers, -- Julien Plissonneau Duquène
Re: Simpler git workflow for packaging with upstreamless repositories
Le 2024-11-29 15:12, Jonas Smedegaard a écrit : Quoting sre4e...@free.fr (2024-11-29 13:29:05) Would it be possible to make it the default when there is already an existing `pristine-tar` branch in the repo being worked on? Please consider filing that as a bugreport against bit-buildpackage - I suspect that has a far better traction than asking speculatively in this mailinglist and assuming that others picks up your thought and does that task for you. That would be a duplicate though, someone else already had that great idea: see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=557606 opened 15 years and a few days ago. Cheers, -- Julien Plissonneau Duquène
Re: Simpler git workflow for packaging with upstreamless repositories
Quoting sre4e...@free.fr (2024-11-29 13:29:05) > Le 2024-11-29 03:17, Otto Kekäläinen a écrit : > > > but right now I warmly recommend people use 'pristine-tar = True' in > > their gbp.confs. > > Would it be possible to make it the default when there is already an > existing `pristine-tar` branch in the repo being worked on? Please consider filing that as a bugreport against bit-buildpackage - I suspect that has a far better traction than asking speculatively in this mailinglist and assuming that others picks up your thought and does that task for you. Kind regards, and thanks for the great idea, - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, Le 2024-11-29 03:17, Otto Kekäläinen a écrit : but right now I warmly recommend people use 'pristine-tar = True' in their gbp.confs. Would it be possible to make it the default when there is already an existing `pristine-tar` branch in the repo being worked on? Cheers, -- Julien Plissonneau Duquène
Re: Simpler git workflow for packaging with upstreamless repositories
2024, നവം 29 7:48:10 AM Otto Kekäläinen :
> Thanks Pirate Praveen for providing the first actual concrete case in
> this thread where pristine-tar had some issue!
>
> I noticed an interesting thing about this case:
>
> ± origtargz --download-only
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
> fatal: ambiguous argument
> '60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}': unknown revision or
> path not in the working tree.
> Use '--' to separate paths from revisions, like this:
> 'git [...] -- [...]'
> fatal: not a valid object name:
> 60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> pristine-tar: command failed: git archive --format=tar
> 60b4383b8c982ac64553f2754abaefe7ca7ebf79\^\{tree\} | (cd
> '/tmp/pristine-tar.obWgetreHi' && tar x)
>
> ± git show 60b4383b8c982ac64553f2754abaefe7ca7ebf79
> fatal: bad object 60b4383b8c982ac64553f2754abaefe7ca7ebf79
>
> ± git fetch origin 60b4383b8c982ac64553f2754abaefe7ca7ebf79
> fatal: remote error: upload-pack: not our ref
> 60b4383b8c982ac64553f2754abaefe7ca7ebf79
> fatal: the remote end hung up unexpectedly
>
> ± gbp export-orig --pristine-tar
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
>
> And then suddenly the git ref has emerged:
>
> ± git show a1567ff8077126b7aa8536b779e3e445ba367a49
> tree a1567ff8077126b7aa8536b779e3e445ba367a49
> .github/
> LICENSE.md
> README.md
> index.js
> package.json
> test/
>
> ± gbp export-orig --pristine-tar
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
> gbp:info: Creating
> /home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
>
> Also comparing output with what I manually downloaded from
> https://github.com/npm/cacache/releases/tag/v17.0.3
> $ sha256sum v17.0.3.tar.gz node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
> 2daa2c943a9cf316eef10eda6883ac967ca32cc28de9decef147ea42bdc34283
> v17.0.3.tar.gz
> 2daa2c943a9cf316eef10eda6883ac967ca32cc28de9decef147ea42bdc34283
> node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
>
> Not sure what happened here. However in the end pristine-tar worked,
> and I was able to use it to verify the tarball. Longer notes in
> https://pad.debian.net/p/node-cacache-pristine-tar.
>
> There are however a lot of oddities in this package that makes it unusual
> - You don't have 'pristine-tar = True' in debian/gbp.conf. You should
> have it to enforce it is used and git pulled and git pushed
> consistently.
> - There is no README.source explaining what this '+~cs10.3.7' thing in
> the version is. I assumed you had repackaged something, but then also
> was surprised that it actuall was the same as upstream.
> - This package consists of the main package and 5 components that are
> all mangled together. Is that necessary? I am surpised such a complex
> thing actually seems to work
This is standard option of uscan documented in uscan man page.
>
> In summary: nothing in this is an argument to stop using pristine-tar
> in all packages. I think Theodore Ts'o also laid out pretty well all
> the benefits of pristine-tar and why it was originally developed, and
> those requirements and benefits still stand. Sure, we can in future
> also have other ways to do this in a future debian package format 3.1,
> but right now I warmly recommend people use 'pristine-tar = True' in
> their gbp.confs.
I only shared my experience of origtargz failing very commonly for me.
Re: Simpler git workflow for packaging with upstreamless repositories
"Theodore Ts'o" writes: > Perhaps we could avoid talking past we formally had a list of > requirements, and then match possible alternative approachs with how > well they meet the agreed-upon requirements, and which requirements > proponents want to dispense with because (at least for them), It's > Just Not Worth It? Yes please! I suspect one root problem here is that people have different conflicting requirements, and everyone primarily relate to their own situation. I often work in offline mode too, but never had any problem with the download tarball approach. After 'git clone' (which require internet) the first thing I normally do is to attempt a build, and git-buildpackage download the *.orig.tar.* automatically for me. Then I leave the tarball around on my laptop and never think about it. It is rare for me to happen to have a git repository of a package around and not have its corresponding tarballs too, but workflows differ. >> If we are worried about malicious upstreams replacing tarballs, or >> man-in-the-middle attacks, I think my debian/upstream/*SUMS approach is >> a more effective solution to that problem. > > Maybe... if there were tools that made it super easy to validate the > tarball against the *SUMS files without needing to unpack the tarball > first? I think 'sha256sum -c mypackage-git-repository/debian/source/SHA256SUM' should work if you have the tarballs in the current directory. > Possibly with an inline GPG signature so we don't have to have > separate SHA256SUM and SHA256SUM.asc files? For bonus points, maybe > also a tool that validates a SHA256SUM file with a git commit id, > again without needing to do a "git checkout" first? > > I will note that this approach would break backwads compatibility with > existing Debian source packaging, right? That is, you're proposing > that the debian/usptream/*SUMS file would replace the > *.orig.tar.gz.asc file? I don't think that works: the nice thing with *.orig.tar.gz.asc is that we get upstream's signature file into Debian, allowing users to follow the audit trail back to upstream. My primary motivation is to make it possible to record under debian/ the intended (by the package maintainer) checksums of the *.orig.tar.* and (when they are different) upstream tarballs. We don't have any way to record that in debian/ today, I think. The only record of this is indirectly with the maintainer signing the *.changes file during package upload. But that is weak (only successfully uploaded packages are protected, not work-in-progress) and not widely audited (*.changes files aren't stored forever, or are they?). /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hello, On Thu 28 Nov 2024 at 06:17pm -08, Otto Kekäläinen wrote: > In summary: nothing in this is an argument to stop using pristine-tar > in all packages. I think Theodore Ts'o also laid out pretty well all > the benefits of pristine-tar and why it was originally developed, and > those requirements and benefits still stand. The very name of the tool is intended as an encouragement for us to move away from tarballs. It's making fun of Debian's attachment to upstream tarballs. -- Sean Whitton
Re: Simpler git workflow for packaging with upstreamless repositories
Thanks Pirate Praveen for providing the first actual concrete case in
this thread where pristine-tar had some issue!
I noticed an interesting thing about this case:
± origtargz --download-only
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
fatal: ambiguous argument
'60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}': unknown revision or
path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git [...] -- [...]'
fatal: not a valid object name: 60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
pristine-tar: command failed: git archive --format=tar
60b4383b8c982ac64553f2754abaefe7ca7ebf79\^\{tree\} | (cd
'/tmp/pristine-tar.obWgetreHi' && tar x)
± git show 60b4383b8c982ac64553f2754abaefe7ca7ebf79
fatal: bad object 60b4383b8c982ac64553f2754abaefe7ca7ebf79
± git fetch origin 60b4383b8c982ac64553f2754abaefe7ca7ebf79
fatal: remote error: upload-pack: not our ref
60b4383b8c982ac64553f2754abaefe7ca7ebf79
fatal: the remote end hung up unexpectedly
± gbp export-orig --pristine-tar
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
And then suddenly the git ref has emerged:
± git show a1567ff8077126b7aa8536b779e3e445ba367a49
tree a1567ff8077126b7aa8536b779e3e445ba367a49
.github/
LICENSE.md
README.md
index.js
package.json
test/
± gbp export-orig --pristine-tar
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
gbp:info: Creating
/home/otto/debian/js-team/node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
Also comparing output with what I manually downloaded from
https://github.com/npm/cacache/releases/tag/v17.0.3
$ sha256sum v17.0.3.tar.gz node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
2daa2c943a9cf316eef10eda6883ac967ca32cc28de9decef147ea42bdc34283 v17.0.3.tar.gz
2daa2c943a9cf316eef10eda6883ac967ca32cc28de9decef147ea42bdc34283
node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
Not sure what happened here. However in the end pristine-tar worked,
and I was able to use it to verify the tarball. Longer notes in
https://pad.debian.net/p/node-cacache-pristine-tar.
There are however a lot of oddities in this package that makes it unusual
- You don't have 'pristine-tar = True' in debian/gbp.conf. You should
have it to enforce it is used and git pulled and git pushed
consistently.
- There is no README.source explaining what this '+~cs10.3.7' thing in
the version is. I assumed you had repackaged something, but then also
was surprised that it actuall was the same as upstream.
- This package consists of the main package and 5 components that are
all mangled together. Is that necessary? I am surpised such a complex
thing actually seems to work
In summary: nothing in this is an argument to stop using pristine-tar
in all packages. I think Theodore Ts'o also laid out pretty well all
the benefits of pristine-tar and why it was originally developed, and
those requirements and benefits still stand. Sure, we can in future
also have other ways to do this in a future debian package format 3.1,
but right now I warmly recommend people use 'pristine-tar = True' in
their gbp.confs.
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 28, 2024 at 10:35:49AM +0100, Simon Josefsson wrote: > > I think this is a good example of us talking past each other in this > thread: some people question the value of pristine in the first place > (and I've been compelled by those arguments), and some people argue that > the cost is small and there are no bugs (or at least lack of bug > reports). Our current system addresses a number of requirements, and it seems to me that a number of the alternate solutions don't necessarily meet all of the requirements. Some of these requirements include: *) We want an easy way to make sure the sources used to rebuild debian packages aren't maliciously modified. We do this today via PGP signed tarballs. *) As much as possible, we want to be able to use the unmodified source files are officially released by upstream. Which might be a tarball and/or a signed git tag. *) The sources that we redistribute alongside our binary packages must be DFSG compliant. In some cases this might mean repacking the tar file, and might interfere with using upstream's official signed git tag. *) We don't want to break the interface provided by "apt-get source" and debian source packages more generally. I have my own personal requirements that might not be shared by others. For example, I don't like having to keep source tarballs around when I need to rebuild debian packages, and tracking them down by hand. I also want to keep the storage overhead as much as possible (hence, why I like pristine-tar). And I want it to all work automatically using my current build tools, which today is git-buildpackage. And finally, I am occasionally doing work in network constrained environments (for example, while using my laptop in an airplane), so I prefer to avoid solutions that start with "and then we download the tar.gz file from the network". Perhaps we could avoid talking past we formally had a list of requirements, and then match possible alternative approachs with how well they meet the agreed-upon requirements, and which requirements proponents want to dispense with because (at least for them), It's Just Not Worth It? > If we are worried about malicious upstreams replacing tarballs, or > man-in-the-middle attacks, I think my debian/upstream/*SUMS approach is > a more effective solution to that problem. Maybe... if there were tools that made it super easy to validate the tarball against the *SUMS files without needing to unpack the tarball first? Possibly with an inline GPG signature so we don't have to have separate SHA256SUM and SHA256SUM.asc files? For bonus points, maybe also a tool that validates a SHA256SUM file with a git commit id, again without needing to do a "git checkout" first? I will note that this approach would break backwads compatibility with existing Debian source packaging, right? That is, you're proposing that the debian/usptream/*SUMS file would replace the *.orig.tar.gz.asc file? - Ted
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 28, 2024 at 07:33:27PM +0530, Pirate Praveen wrote: > 2024, നവം 28 5:46:47 PM Andrey Rakhmatullin : > > > >> Same works with > >> > >> pravi@mahishi2:/tmp/node-cacache$ gbp export-orig --pristine-tar > > > > Not sure what does this imply. > > gbp is able to recreate the orig tars when origtargz fails. So it is a known > work around. And if both call pristine-tar for this, I wonder what's the difference in arguments (or what else could be different there?). -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
2024, നവം 28 5:46:47 PM Andrey Rakhmatullin : > >> Same works with >> >> pravi@mahishi2:/tmp/node-cacache$ gbp export-orig --pristine-tar > > Not sure what does this imply. gbp is able to recreate the orig tars when origtargz fails. So it is a known work around.
Re: Simpler git workflow for packaging with upstreamless repositories
* Andrey Rakhmatullin [241128 11:06]: > On Thu, Nov 28, 2024 at 10:49:09AM +0100, gregor herrmann wrote: > > > > As person B I don't want to go hunting for a tarball and place it in > > > > the right place, I want gbp to re-create it from the pristine-tar > > > > branch. > > > Who cares? gbp export-orig takes care of it, unless you need to actually > > > upload it to the archive. Just make sure to never use the upstream tarball then, given it won't contain the same files. In your case where you don't start with the upstream tarball it's -probably- fine, as long as nobody then ventures into "use uscan"-land. > > Often I want to upload it to the archive, that's quite typical in > > teams: A uploads -1 and B uploads -2 with a bugfix or whatever. > > The hypothetical change in the tools to silently download the tarball from > the archive will make this easy. Obviously these hyptothetical tools will also deal with all previously uploaded versions correctly. SCNR, Chris
Re: Simpler git workflow for packaging with upstreamless repositories
* Paul Gevers [241128 12:52]: > Hi, > > On 11/28/24 10:56, Pirate Praveen wrote: > > pristine-tar almost always fail when there are multiple orig tar files. > > I did not report bugs since the limitation of not working 100% is a > > known issue already. > > I'm surprised to hear this. src:cacti is using multiple orig tar files (2) > for several years now and I've been using pristine-tar (via gbp) to store > them. But can you create the tarballs from that repo? cacti $ git show pristine-tar commit 1a42ec3d9772f93caa8b98cb78c5b0d2b0b320e9 (origin/pristine-tar, pristine-tar) Author: Paul Gevers AuthorDate: Wed Oct 9 14:04:17 2024 +0200 Commit: Paul Gevers CommitDate: Wed Oct 9 14:04:17 2024 +0200 pristine-tar data for cacti_1.2.28+ds1.orig.tar.gz diff --git a/cacti_1.2.28+ds1.orig.tar.gz.delta b/cacti_1.2.28+ds1.orig.tar.gz.delta new file mode 100644 index ..382fe265 Binary files /dev/null and b/cacti_1.2.28+ds1.orig.tar.gz.delta differ diff --git a/cacti_1.2.28+ds1.orig.tar.gz.id b/cacti_1.2.28+ds1.orig.tar.gz.id new file mode 100644 index ..1fa49d3d --- /dev/null +++ b/cacti_1.2.28+ds1.orig.tar.gz.id @@ -0,0 +1 @@ +979cace8c3b2467cdc418b849810b0ec2a54c7ae cacti $ git show 979cace8c3b2467cdc418b849810b0ec2a54c7ae fatal: bad object 979cace8c3b2467cdc418b849810b0ec2a54c7ae ... I'll note that the tag upstream/1.2.28+ds1 points to 6aa33682cc62c3fbb638ff85dc02bf531406de93 . C.
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 28, 2024 at 05:37:41PM +0530, Pirate Praveen wrote:
> This can now be reproduced with node-cacache
>
> pravi@mahishi2:/tmp$ gbp clone --pristine-tar
> g...@salsa.debian.org:js-team/node-cacache.git
> gbp:info: Cloning from 'g...@salsa.debian.org:js-team/node-cacache.git'
> pravi@mahishi2:/tmp$ cd node-cacache/
> pravi@mahishi2:/tmp/node-cacache$ less debian/gbp.conf
> pravi@mahishi2:/tmp/node-cacache$ origtargz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
> pristine-tar: successfully generated
> ../node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
> fatal: ambiguous argument '60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}':
> unknown revision or path not in the working tree.
node-cacache_17.0.3+~cs10.3.7.orig.tar.gz.id contains that hash and there
is indeed no object with that has in that repo. So I guess the main
question is how was this repo generated.
> Same works with
>
> pravi@mahishi2:/tmp/node-cacache$ gbp export-orig --pristine-tar
Not sure what does this imply.
--
WBR, wRAR
signature.asc
Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On 11/28/24 5:29 PM, Pirate Praveen wrote:
On 11/28/24 5:22 PM, Paul Gevers wrote:
Hi,
On 11/28/24 10:56, Pirate Praveen wrote:
pristine-tar almost always fail when there are multiple orig tar
files. I did not report bugs since the limitation of not working 100%
is a known issue already.
I'm surprised to hear this. src:cacti is using multiple orig tar files
(2) for several years now and I've been using pristine-tar (via gbp)
to store them.
Do you import them using gbp import-orig --pristine-tar as well? or just
pristine-tar commit?
Paul
https://salsa.debian.org/debian/cacti/-/tree/pristine-tar?ref_type=heads
This can now be reproduced with node-cacache
pravi@mahishi2:/tmp$ gbp clone --pristine-tar
g...@salsa.debian.org:js-team/node-cacache.git
gbp:info: Cloning from 'g...@salsa.debian.org:js-team/node-cacache.git'
pravi@mahishi2:/tmp$ cd node-cacache/
pravi@mahishi2:/tmp/node-cacache$ less debian/gbp.conf
pravi@mahishi2:/tmp/node-cacache$ origtargz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
pristine-tar: successfully generated
../node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
fatal: ambiguous argument
'60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}': unknown revision or
path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git [...] -- [...]'
fatal: not a valid object name:
60b4383b8c982ac64553f2754abaefe7ca7ebf79^{tree}
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
pristine-tar: command failed: git archive --format=tar
60b4383b8c982ac64553f2754abaefe7ca7ebf79\^\{tree\} | (cd
'/tmp/pristine-tar.nbS9WwWNUi' && tar x)
debian/gbp.conf which defines the extra orig tars,
[DEFAULT]
component=['fs-minipass', 'infer-owner', 'npmcli-move-file',
'npmcli-fs', 'gar-promisify']
[import-orig]
filter=.gitignore
Then this is imported with gbp import-orig --pristine-tar
Same works with
pravi@mahishi2:/tmp/node-cacache$ gbp export-orig --pristine-tar
gbp:info: Creating /tmp/node-cacache_17.0.3+~cs10.3.7.orig.tar.gz
gbp:info: Creating
/tmp/node-cacache_17.0.3+~cs10.3.7.orig-fs-minipass.tar.gz
gbp:info: Creating
/tmp/node-cacache_17.0.3+~cs10.3.7.orig-infer-owner.tar.gz
gbp:info: Creating
/tmp/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-move-file.tar.gz
gbp:info: Creating /tmp/node-cacache_17.0.3+~cs10.3.7.orig-npmcli-fs.tar.gz
gbp:info: Creating
/tmp/node-cacache_17.0.3+~cs10.3.7.orig-gar-promisify.tar.gz
Re: Simpler git workflow for packaging with upstreamless repositories
On 11/28/24 5:22 PM, Paul Gevers wrote: Hi, On 11/28/24 10:56, Pirate Praveen wrote: pristine-tar almost always fail when there are multiple orig tar files. I did not report bugs since the limitation of not working 100% is a known issue already. I'm surprised to hear this. src:cacti is using multiple orig tar files (2) for several years now and I've been using pristine-tar (via gbp) to store them. Do you import them using gbp import-orig --pristine-tar as well? or just pristine-tar commit? Paul https://salsa.debian.org/debian/cacti/-/tree/pristine-tar?ref_type=heads
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, On 11/28/24 10:56, Pirate Praveen wrote: pristine-tar almost always fail when there are multiple orig tar files. I did not report bugs since the limitation of not working 100% is a known issue already. I'm surprised to hear this. src:cacti is using multiple orig tar files (2) for several years now and I've been using pristine-tar (via gbp) to store them. Paul https://salsa.debian.org/debian/cacti/-/tree/pristine-tar?ref_type=heads
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, 28 Nov 2024 15:06:15 +0500, Andrey Rakhmatullin wrote: > > Often I want to upload it to the archive, that's quite typical in > > teams: A uploads -1 and B uploads -2 with a bugfix or whatever. > The hypothetical change in the tools to silently download the tarball from > the archive will make this easy. Oh, sure, and I'm not wedded to pristine-tar or gbp using the pristine-tar branch. I just want to say that working in teams requires easy access to the identical .orig.tar.*, and "easy" for me also means "automatic" (in contrast to "download from somewhere and put it in some directory manually"). Cheers, gregor -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- BOFH excuse #399: We are a 100% Microsoft Shop.
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 28, 2024 at 10:49:09AM +0100, gregor herrmann wrote: > > > As person B I don't want to go hunting for a tarball and place it in > > > the right place, I want gbp to re-create it from the pristine-tar > > > branch. > > Who cares? gbp export-orig takes care of it, unless you need to actually > > upload it to the archive. > > Often I want to upload it to the archive, that's quite typical in > teams: A uploads -1 and B uploads -2 with a bugfix or whatever. The hypothetical change in the tools to silently download the tarball from the archive will make this easy. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
2024, നവം 28 3:06:13 PM Simon Josefsson : > I think this is a good example of us talking past each other in this > thread: some people question the value of pristine in the first place > (and I've been compelled by those arguments), and some people argue that > the cost is small and there are no bugs (or at least lack of bug > reports). > pristine-tar almost always fail when there are multiple orig tar files. I did not report bugs since the limitation of not working 100% is a known issue already. Many packages in js-team and gitlab package is an example. In some cases when origtargz command fails, gbp export-orig --pristine-tar works. Again this incompatibility between using pristine-tar directly and gbp export-orig seems like a known issue. A simple fix would probably for origtargz to fall back to gbp export-orig if pristine-tar fail.
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, 28 Nov 2024 09:41:53 +0100, Marco d'Itri wrote: > On Nov 27, gregor herrmann wrote: > > As person B I don't want to go hunting for a tarball and place it in > > the right place, I want gbp to re-create it from the pristine-tar > > branch. > Who cares? gbp export-orig takes care of it, unless you need to actually > upload it to the archive. Often I want to upload it to the archive, that's quite typical in teams: A uploads -1 and B uploads -2 with a bugfix or whatever. Cheers, gregor -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- BOFH excuse #229: wrong polarity of neutron flow
Re: Simpler git workflow for packaging with upstreamless repositories
"Theodore Ts'o" writes: > On Tue, Nov 26, 2024 at 04:27:37PM +0100, Simon Josefsson wrote: >> I have never understood what value there is in duplicating the uploaded >> tarball in the git repository. > > The actual cost of storing the pristine tarball is quite small. I think this is a good example of us talking past each other in this thread: some people question the value of pristine in the first place (and I've been compelled by those arguments), and some people argue that the cost is small and there are no bugs (or at least lack of bug reports). > For example: > > commit 91c7ab39337da63371b4814bef2b2aaf85a9e37c (origin/pristine-tar, > pristine-tar) > Author: Theodore Ts'o > Date: Mon May 20 23:12:54 2024 -0400 > > pristine-tar data for e2fsprogs_1.47.1.orig.tar.gz > > e2fsprogs_1.47.1.orig.tar.gz.asc | 11 +++ > e2fsprogs_1.47.1.orig.tar.gz.delta | Bin 0 -> 63961 bytes > e2fsprogs_1.47.1.orig.tar.gz.id| 1 + > 3 files changed, 12 insertions(+) > > Compare if I had to keep all of the old release tarballs around: > > 9.5M e2fsprogs-1.47.1.tar.gz > > The reason why I find pristine-tar *super* valuable is because it > stashes the signed tarball and tarball in a highly efficient way, and > which can be easily backed up by just doing a "git push" to github / > git.kernel.org / salsa. I can then just kick off a git-buildpackage > in a super-convenient way, so the tooling is quite mature and > convenient for development velocity. > > I could imagine an alternate way of generating data for > git-buildpackage, by replacing the pristine with something that stores > the detached GPG signature, and then a shell script which generates > the orig.tar.gz, for example at [1]. But now we'd have third-party > users who want to rebuild the debian packages from source executing an > arbitrary shell script found in the git repository to generate the > orig.tar.gz file, which would be a security nightmare. Pristine-tar > is a much better from that perspective. > > [1] https://github.com/tytso/e2fsprogs/blob/master/util/gen-git-tarball Yeah, this is nice, but I appear to have all of that with git-pbuildpackage, uscan, origtargz etc downloading the upstream tarball automatically already today. If we are worried about malicious upstreams replacing tarballs, or man-in-the-middle attacks, I think my debian/upstream/*SUMS approach is a more effective solution to that problem. Pristine-tar seems like a tool-centric solution that isn't used elsewhere in the FOSS ecosystem. Hash checksums are widely used to solve the security concerns, and people know about those concepts even without learning anything about Debian let alone git-buildpackage or pristine-tar. If we are worried about upstreams going away so the tarball URLs doesn't work, I like the Guix approach to 1) store hash checksums and 2) a mirror system that fall back to the Software Heritage. That also uses known established concepts (SHA256 hashes + URL list) to solve the problem, without having to learn git-buildpackage or pristine-tar. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 28, 2024 at 09:41:53AM +0100, Marco d'Itri wrote: > On Nov 27, gregor herrmann wrote: > > > As person B I don't want to go hunting for a tarball and place it in > > the right place, I want gbp to re-create it from the pristine-tar > > branch. > Who cares? gbp export-orig takes care of it, unless you need to actually > upload it to the archive. (only if the file contents inside those are identical) -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Nov 27, gregor herrmann wrote: > As person B I don't want to go hunting for a tarball and place it in > the right place, I want gbp to re-create it from the pristine-tar > branch. Who cares? gbp export-orig takes care of it, unless you need to actually upload it to the archive. -- ciao, Marco signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 04:27:37PM +0100, Simon Josefsson wrote: > I have never understood what value there is in duplicating the uploaded > tarball in the git repository. The actual cost of storing the pristine tarball is quite small. For example: commit 91c7ab39337da63371b4814bef2b2aaf85a9e37c (origin/pristine-tar, pristine-tar) Author: Theodore Ts'o Date: Mon May 20 23:12:54 2024 -0400 pristine-tar data for e2fsprogs_1.47.1.orig.tar.gz e2fsprogs_1.47.1.orig.tar.gz.asc | 11 +++ e2fsprogs_1.47.1.orig.tar.gz.delta | Bin 0 -> 63961 bytes e2fsprogs_1.47.1.orig.tar.gz.id| 1 + 3 files changed, 12 insertions(+) Compare if I had to keep all of the old release tarballs around: 9.5M e2fsprogs-1.47.1.tar.gz The reason why I find pristine-tar *super* valuable is because it stashes the signed tarball and tarball in a highly efficient way, and which can be easily backed up by just doing a "git push" to github / git.kernel.org / salsa. I can then just kick off a git-buildpackage in a super-convenient way, so the tooling is quite mature and convenient for development velocity. I could imagine an alternate way of generating data for git-buildpackage, by replacing the pristine with something that stores the detached GPG signature, and then a shell script which generates the orig.tar.gz, for example at [1]. But now we'd have third-party users who want to rebuild the debian packages from source executing an arbitrary shell script found in the git repository to generate the orig.tar.gz file, which would be a security nightmare. Pristine-tar is a much better from that perspective. [1] https://github.com/tytso/e2fsprogs/blob/master/util/gen-git-tarball Cheers, - Ted
Re: Simpler git workflow for packaging with upstreamless repositories
* Marc Haber [241127 13:52]: > On Wed, Nov 27, 2024 at 01:48:33PM +0100, Chris Hofstaedtler wrote: > > * Marc Haber [241127 13:28]: > > > On Wed, Nov 27, 2024 at 04:58:00PM +0500, Andrey Rakhmatullin wrote: > > > > Yup, as I said it makes sense. It just feels fragile to me when the > > > > "pristine" tarball for a given upstream tag in a given repo is not > > > > determined until someone uploads it. > > > > > > I would love to have a possibility to just push a new upstream tarball > > > into the archive at the very beginning of the process of packaging the > > > new upstream version (and without triggering anything else in Debian). > > > > I hear what you're saying, but in the end it is still a workaround, > > right? > > It's just "putting a nail" into a new upstream version. > > > Just uploading a new tarball with every debian revision seems so > > much more straightforward. > > That means that the upstream tarball is possible to get wrong with any > new upload. I am not sure whether I like that. At least for some packages, what I consider the interesting part is the contents of the "upstream" branch, and not what is in upstream's tarball. > I might motivate me, > though, to finally explore which of the tools I am holding wrong with > upstream tarball handling. Reconciling these things seems like a lot of busywork, as you said. While I haven't given up on pristine-tar yet, this thread pushes my motivation towards exploring the +dfsgN-1 idea "fo' real". Chris
Re: Simpler git workflow for packaging with upstreamless repositories
On Wed, Nov 27, 2024 at 01:48:33PM +0100, Chris Hofstaedtler wrote: > * Marc Haber [241127 13:28]: > > On Wed, Nov 27, 2024 at 04:58:00PM +0500, Andrey Rakhmatullin wrote: > > > Yup, as I said it makes sense. It just feels fragile to me when the > > > "pristine" tarball for a given upstream tag in a given repo is not > > > determined until someone uploads it. > > > > I would love to have a possibility to just push a new upstream tarball > > into the archive at the very beginning of the process of packaging the > > new upstream version (and without triggering anything else in Debian). > > I hear what you're saying, but in the end it is still a workaround, > right? It's just "putting a nail" into a new upstream version. > Just uploading a new tarball with every debian revision seems so > much more straightforward. That means that the upstream tarball is possible to get wrong with any new upload. I am not sure whether I like that. I might motivate me, though, to finally explore which of the tools I am holding wrong with upstream tarball handling. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Re: Simpler git workflow for packaging with upstreamless repositories
* Marc Haber [241127 13:28]: > On Wed, Nov 27, 2024 at 04:58:00PM +0500, Andrey Rakhmatullin wrote: > > Yup, as I said it makes sense. It just feels fragile to me when the > > "pristine" tarball for a given upstream tag in a given repo is not > > determined until someone uploads it. > > I would love to have a possibility to just push a new upstream tarball > into the archive at the very beginning of the process of packaging the > new upstream version (and without triggering anything else in Debian). I hear what you're saying, but in the end it is still a workaround, right? Just uploading a new tarball with every debian revision seems so much more straightforward. I know, some packages might have huge tarballs. On the other hand, some packages like src:linux seem to get a new tarball each time anyway (because often there's no -2?). I imagine each package can do this today, by producing an +dfsg-1 version for each upload ... Chris
Re: Simpler git workflow for packaging with upstreamless repositories
On Wed, Nov 27, 2024 at 04:58:00PM +0500, Andrey Rakhmatullin wrote: > Yup, as I said it makes sense. It just feels fragile to me when the > "pristine" tarball for a given upstream tag in a given repo is not > determined until someone uploads it. I would love to have a possibility to just push a new upstream tarball into the archive at the very beginning of the process of packaging the new upstream version (and without triggering anything else in Debian). I have done -0 experimental uploads just for the sake of the orig.tar.gz. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 08:30:31PM -0800, Otto Kekäläinen wrote: > > > > One possible rebuttal to this is "gbp needs to do the right thing then". > > > > Currently gbp by default generates a broken tarball, which is also a > > > > source of confusion for many. > > > > > > Do you have a bug report number? > > > > No. > > I've found #902249 which is titled "Pull tarballs from the archive (or > > upstream location)", maybe that's the one you think about. I haven't read > > it, except for the "I hoped we could stay out of that business in 2018 but > > since tarballs are still _the_ _thing_ in Debian" line, which I liked. > > > > For the avoidance of doubt, I don't think gbp *can* do the right thing > > there. It's not my rebuttal. Maybe gbp should just refuse to generate a > > random tarball from a commit-ish and let^Wrequire people to provide one or > > provide a way to generate one in a correct way. > > I often hear this complaint about pristine-tar, but I don't take it > seriously until somebody actually files a bug report and has a > reproducible case. For every single package I maintain I use > pristine-tar and it works correctly. Maybe you (and maybe some other people too?) misunderstood what I meant. I meant that if the repo doesn't enable pristine-tar via d/gbp.conf and you didn't enable it globally on your system then by default building that repo will generate and use a tarball that is not identical to the one in the archive. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On 27/11/24 04:30, Otto Kekäläinen wrote: Secondly, it is perfectly valid for evey single package to have a debian/gbp.conf and I would in fact prefer that. For every upstream we need to have metadata on: - do they have tarball releases (pristine-tar true/false) - do they have git tags and what is the format (upstream-vcs-tag) - are those git tags expected to be signed or not (upstream-signatures on/off) It's great that these important pieces of information can be stored in d/gbp.conf (and I will keep doing it), but this information should really be stored somewhere in d/upstream/metadata. On the same topic, many people have asked in the past for a machine-readable and tooling-independent way to specify: a) information about the way upstream does releases and, b) the packaging workflow used by that package manager. For some, this documentation would be a step-stop towards a standardized workflow. For others, it would act as a replacement for a standardized workflow. We should really start a discussion on a "formalized README.source.md". Regards, -- Gioele Barabucci
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 08:49:44PM +0100, Simon Josefsson wrote: > >> > >> > Yes, as they don't enable pristine-tar > >> > >> > >> > >> Is pristine-tar still valuable these days? > >> > > > >> > > Unfortunately yes. AFAIK the two options for fixing this that are > >> > > usually proposed are: > >> > > > >> > > 1) treat it as a problem of each individual developer, just like > >> > > pristine-tar. Instead of pristine-tar, invent new tooling to manage > >> > > tarballs. > >> > > This path often tries to solve the problem only for Debian and only > >> > > in a narrow scenario. > >> > > > >> > > 2) Have all uploads always supply a new orig.tar.gz. This could mean > >> > > either treating every package as Debian-native, or some other > >> > > solution. > >> > > This is a global solution and reduces complexity instead of adding > >> > > to it. > >> > > >> > Until we record expected upstream tarball hashes in a debian/* file, an > >> > acceptable approach seems to be to skip the pristine-tar branch and be > >> > sure to download the previous orig.tar.* + orig.tar.*.asc from the > >> > Debian archive, instead of attempting to re-generate it from the > >> > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). > >> > >> This is 1). It cannot be done generically as it requires knowing > >> where to download from, etc. > > > > The archive, when the tarball is already there. > > > > These suggestions never discuss what to do when the tarball was never > > uploaded yet, even I didn't discuss that for simplicity. It makes sense > > from some PoVs, at least when one doesn't use pristine-tar to make a > > tarball that has differences in the actual content, not just bit > > differences in the tarball itself while have identical file contents. > > If you haven't made an upload, then wouldn't you have the tarball > locally while working on preparing the upload? > > And if someone doesn't have the orig.tar.gz locally, then why would > anyone want to get it from a random git repository, rather than fetching > it from the Debian archive or from upstream's release page? What is the > use-case here that am I missing? Yup, as I said it makes sense. It just feels fragile to me when the "pristine" tarball for a given upstream tag in a given repo is not determined until someone uploads it. And, as I also said, there are use cases (arguably buggy) when the tarball contents is actually modified. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 08:49:44PM +0100, Simon Josefsson wrote: > Andrey Rakhmatullin writes: > > On Tue, Nov 26, 2024 at 06:54:18PM +0100, Chris Hofstaedtler wrote: > >> This is 1). It cannot be done generically as it requires knowing > >> where to download from, etc. > > > > The archive, when the tarball is already there. > > > > These suggestions never discuss what to do when the tarball was never > > uploaded yet, even I didn't discuss that for simplicity. It makes sense > > from some PoVs, at least when one doesn't use pristine-tar to make a > > tarball that has differences in the actual content, not just bit > > differences in the tarball itself while have identical file contents. > > If you haven't made an upload, then wouldn't you have the tarball > locally while working on preparing the upload? > > And if someone doesn't have the orig.tar.gz locally, then why would > anyone want to get it from a random git repository, rather than fetching > it from the Debian archive or from upstream's release page? What is the > use-case here that am I missing? .orig.tar tarball handling is a horrible mess. I have been a DD for two decades, have experienced at least three methods to handle them, and still, upstream tarballs get put by tools in a place where other tools don't expect them, get misnamed, are compressed by one tool in a format that the next tool doesn't understand (yet/any more). It invariably always takes me at least a second try to get it right, and I would LOVE that having a pristine-tar branch would make me get rid of this chaos by just rebuilding the .orig.tar.gz if it isnt found in a place where $TOOL expects it. It's only a major nuisance compared to others that we have in the project, but if I had an Euro for every time I have sighed and mv'ed a tarball, I'd be rich and be able to work almost full time on Debian. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Re: Simpler git workflow for packaging with upstreamless repositories
gregor herrmann writes: > On Tue, 26 Nov 2024 20:49:44 +0100, Simon Josefsson wrote: > >> If you haven't made an upload, then wouldn't you have the tarball >> locally while working on preparing the upload? >> And if someone doesn't have the orig.tar.gz locally, then why would >> anyone want to get it from a random git repository, rather than fetching >> it from the Debian archive or from upstream's release page? What is the >> use-case here that am I missing? > > Person A creates a package, person B wants to work on it; use cases: > teams in general, and sponsoring within or outside of teams. > > As person B I don't want to go hunting for a tarball and place it in > the right place, I want gbp to re-create it from the pristine-tar > branch. That's not terribly convincing for a project-wide default, as maintaining the tarball costs human time and storage space, and any failures in pristine-tar handling takes time to debug too. As person B above, I would personally prefer to chase down the trusted intended tarball from upstream than to trust a random git repository for it. So the workflow you describe is not universal. Few people seems to PGP sign their pristine-tar commits. There are no PGP-signed git tags on pristine-tar branch. So there is no trust or integrity protection of the pristine-tar tarball content, beyond git HTTPS/SSH protection. It is not uncommon to find pristine-tar branches with *.orig.tar.gz but the corresponding (and uploaded) *.orig.tar.gz.asc file is not included. Interestingly, I preferred use of pristine-tar branches before this thread, but unless some more convincing argument appears I have changed my opinion. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, 26 Nov 2024 20:49:44 +0100, Simon Josefsson wrote: > If you haven't made an upload, then wouldn't you have the tarball > locally while working on preparing the upload? > And if someone doesn't have the orig.tar.gz locally, then why would > anyone want to get it from a random git repository, rather than fetching > it from the Debian archive or from upstream's release page? What is the > use-case here that am I missing? Person A creates a package, person B wants to work on it; use cases: teams in general, and sponsoring within or outside of teams. As person B I don't want to go hunting for a tarball and place it in the right place, I want gbp to re-create it from the pristine-tar branch. Cheers, gregor -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- signature.asc Description: Digital Signature
Re: Simpler git workflow for packaging with upstreamless repositories
* Colin Watson [241126 23:34]: > On Tue, Nov 26, 2024 at 09:25:12PM +0100, Simon Josefsson wrote: > > Colin Watson writes: > > > CI for a new-upstream-release commit you've pushed but haven't uploaded > > > to the Debian archive yet, because you're waiting for CI. > > > > > > pristine-tar certainly isn't the only way here (it's true that CI could > > > in principle fetch it directly from upstream), but I find it much easier > > > than the alternatives. > > > > I often wait for CI before making an upload of a new upstream release, > > even without pristine-tar, and Salsa CI hasn't had trouble automatically > > finding a orig.tar.gz to work with for me. So maybe there is some > > additional assumption for your situation? > > Maybe it runs uscan or something in that case; It runs gbp export-orig (plus some fallbacks). If there is no pristine-tar branch, gbp will use the upstream branch. That won't be bit-identical to the upstream tarball. So, without pristine-tar, salsa CI tests something that might not work. > Still, as long as it's building source packages as part of the > process, conceptually I prefer it having all the information in the > repository rather than having to go out to an upstream site that might > be unreliable. Yeah. Chris
Re: Simpler git workflow for packaging with upstreamless repositories
Hi! On Wed, 27 Nov 2024 at 00:47, wrote: > > Hi Johannes, > > Le 2024-11-22 12:45, Johannes Schauer Marin Rodrigues a écrit : > > That's what I'm doing. But that works with tarballs not with upstream > > as git. > > If upstream (deliberately, so this will not change) has DSFG-non-free > > content > > in it, then I should not copy that into a git packaging repo that is > > targeting > > main. Removing the problematic parts from the upstream git repos would > > rewrite > > their history, invalidate tags etc, so the result would not be very > > useful > > anymore. Thus, I usually have one directory on my PC with the upstream > > git and > > another with my Debian packaging git. > > For most projects but the larger ones, you could simply add an > `upstream` remote to your local packaging repo. This makes it easier to > diff or look/import specific upstream commits. > > For packaging I usually work with 3 remotes (and no `origin` to avoid > mistakes): > - debian (the team-managed packaging repo on salsa) > - clone (my own clone of the packaging repo as I don't have update > rights on team repos) > - upstream, which is very convenient for their tags, history and cherry > picking. Note that is your package has the debian/upstream/metadata file, and that file has field "Repository", you can simply run: gbp clone vcsgit: --add-upstream-vcs ..and the resulting repo will automatically have salsa as 'origin' and the real upstream as 'upstreamvcs': ± git remote -v origin g...@salsa.debian.org:debian/entr.git (fetch) origin g...@salsa.debian.org:debian/entr.git (push) upstreamvcs https://github.com/eradman/entr (fetch) upstreamvcs https://github.com/eradman/entr (push) I recommend using 'upstreamvcs' as the upstream origin name to avoid conflicts with tags and branches that start with 'upstream/*' (gbp uses that name automatically).
Re: Simpler git workflow for packaging with upstreamless repositories
Hi Johannes, Le 2024-11-22 12:45, Johannes Schauer Marin Rodrigues a écrit : That's what I'm doing. But that works with tarballs not with upstream as git. If upstream (deliberately, so this will not change) has DSFG-non-free content in it, then I should not copy that into a git packaging repo that is targeting main. Removing the problematic parts from the upstream git repos would rewrite their history, invalidate tags etc, so the result would not be very useful anymore. Thus, I usually have one directory on my PC with the upstream git and another with my Debian packaging git. For most projects but the larger ones, you could simply add an `upstream` remote to your local packaging repo. This makes it easier to diff or look/import specific upstream commits. For packaging I usually work with 3 remotes (and no `origin` to avoid mistakes): - debian (the team-managed packaging repo on salsa) - clone (my own clone of the packaging repo as I don't have update rights on team repos) - upstream, which is very convenient for their tags, history and cherry picking. Cheers, -- Julien Plissonneau Duquène
Re: Simpler git workflow for packaging with upstreamless repositories
On Nov 27, Johannes Schauer Marin Rodrigues wrote: > If the upstream tarball is in the archive, it's probably okay to retrieve one > from there. It's not always trivial because now you need to have the right > "deb-src" lines enabled in your apt sources.list but it's possible. But how do It is trivial enough if I have deb-src configured, or else I spend 30 seconds manually downloading the file. > you collaborate with others on packages that were not uploaded to the desired > suite yet? Do you not use git for collaboration until the package has passed > NEW? I am not sure why I would need a matching .orig.tar.xz for a package which is not in the archive, but in that case if anybody needed one then they could get it along with the binary packages from the temporary directory on my web site. I do not think that you are making a great argument for pristine-tar if it boils down to such an infrequent use case. -- ciao, Marco signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 10:31 PM Otto Kekäläinen wrote: > > Hi, > > (replying in one email to various comments in past 24h) > > > How common debian/gbp.conf points at something else: perhaps gbp's > > defaults are not good, if that many packages need to override them. > > First of all may I ask you to not use terms like 'not good' as it may > come off negative towards the maintainer of that software. Guido has > done an amazing job with git-buildpackage and we certainly want him to > continue iterating on it. > > I also suggest you to participate in gbp development, as currently it > is 95% Guido alone. At > https://salsa.debian.org/agx/git-buildpackage/-/commits/master you can > for example suggest changes to those defaults along with a migration > path, or you can help with just improving the documentation so people > more easily end up choosing the right options. > > Secondly, it is perfectly valid for evey single package to have a > debian/gbp.conf and I would in fact prefer that. For every upstream we > need to have metadata on: > - do they have tarball releases (pristine-tar true/false) > - do they have git tags and what is the format (upstream-vcs-tag) > - are those git tags expected to be signed or not (upstream-signatures on/off) I'm not really following this thread, but I sincerely hope you aren't suggesting that every package in Debian have a gbp.conf file. (I don't think you are on closer inspection, I think you mean that it's valid for every package *that uses gbp* to have a config file.) I very intentionally don't use it (it has its place, but for the packages I work on it is overkill and greatly complicates my job as a developer with very little payoff), and have no desire to use it. If it was to be made mandatory for all packages, in all likelihood I would comply, but I would not be happy. This isn't to lambast gbp at all - like I said, it has its uses. It just doesn't have uses for me, due to a mixture of my needs and my preferences. (Off-topic, but part of the reason I don't use gbp is the docs for how to use it are just not enough to actually figure out what you're doing. Just a documentation revamp or initial tutorial for it would be awesome. Maybe if I ever relearn it I should help with that...) -- Aaron > If a package maintains stable releases or backports or Ubuntu > versions, or if upstream has multiple lines of parallel major version > releases each branch also needs a debian/gbp.conf to tell what are the > correct settings for that branch. > > > > Until we record expected upstream tarball hashes in a debian/* file, an > > > acceptable approach seems to be to skip the pristine-tar branch and be > > > sure to download the previous orig.tar.* + orig.tar.*.asc from the > > > Debian archive, instead of attempting to re-generate it from the > > > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). > > > > This is 1). It cannot be done generically as it requires knowing > > where to download from, etc. > > > > > I have never understood what value there is in duplicating the uploaded > > > tarball in the git repository. Recording a hash of it is sufficient. > > > > The hash is sufficient for knowing it changed, but you still have to > > get the actual tarball from somewhere. > > Don't we have uscan in all packages? If would be nice if > https://codesearch.debian.net/ supported searching just the existense > of files so one could check this out quickly. > > The way most other Linux distros do this is that they have sha256 sums > in the package definition, and a url where to download upstream > tarball from. In Debian we have debian/watch for the latter, but no > sha256sum to verify the exact version. > > We could, if we want, introduce for example a new file > debian/source/origin which could automatically be updated by uscan to > have the exact url and sha256sum and the maintainer just needs to > commit it after running uscan to import the upstream sources. > > However it will take years to design and agree on a new file and have > it in the policy and roll it out. > > The good thing with git-buildpackage is that we have it already, > thousands of packages uses it, and it works well and is fully capable > of producing the source files consumed by dpkg-buildpackage etc to > build the package. > > > > > > One possible rebuttal to this is "gbp needs to do the right thing then". > > > > Currently gbp by default generates a broken tarball, which is also a > > > > source of confusion for many. > > > > > > Do you have a bug report number? > > > > No. > > I've found #902249 which is titled "Pull tarballs from the archive (or > > upstream location)", maybe that's the one you think about. I haven't read > > it, except for the "I hoped we could stay out of that business in 2018 but > > since tarballs are still _the_ _thing_ in Debian" line, which I liked. > > > > For the avoidance of doubt, I don't think gbp *can* do the right thing > > there. It's not my rebuttal. Maybe
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, (replying in one email to various comments in past 24h) > How common debian/gbp.conf points at something else: perhaps gbp's > defaults are not good, if that many packages need to override them. First of all may I ask you to not use terms like 'not good' as it may come off negative towards the maintainer of that software. Guido has done an amazing job with git-buildpackage and we certainly want him to continue iterating on it. I also suggest you to participate in gbp development, as currently it is 95% Guido alone. At https://salsa.debian.org/agx/git-buildpackage/-/commits/master you can for example suggest changes to those defaults along with a migration path, or you can help with just improving the documentation so people more easily end up choosing the right options. Secondly, it is perfectly valid for evey single package to have a debian/gbp.conf and I would in fact prefer that. For every upstream we need to have metadata on: - do they have tarball releases (pristine-tar true/false) - do they have git tags and what is the format (upstream-vcs-tag) - are those git tags expected to be signed or not (upstream-signatures on/off) If a package maintains stable releases or backports or Ubuntu versions, or if upstream has multiple lines of parallel major version releases each branch also needs a debian/gbp.conf to tell what are the correct settings for that branch. > > Until we record expected upstream tarball hashes in a debian/* file, an > > acceptable approach seems to be to skip the pristine-tar branch and be > > sure to download the previous orig.tar.* + orig.tar.*.asc from the > > Debian archive, instead of attempting to re-generate it from the > > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). > > This is 1). It cannot be done generically as it requires knowing > where to download from, etc. > > > I have never understood what value there is in duplicating the uploaded > > tarball in the git repository. Recording a hash of it is sufficient. > > The hash is sufficient for knowing it changed, but you still have to > get the actual tarball from somewhere. Don't we have uscan in all packages? If would be nice if https://codesearch.debian.net/ supported searching just the existense of files so one could check this out quickly. The way most other Linux distros do this is that they have sha256 sums in the package definition, and a url where to download upstream tarball from. In Debian we have debian/watch for the latter, but no sha256sum to verify the exact version. We could, if we want, introduce for example a new file debian/source/origin which could automatically be updated by uscan to have the exact url and sha256sum and the maintainer just needs to commit it after running uscan to import the upstream sources. However it will take years to design and agree on a new file and have it in the policy and roll it out. The good thing with git-buildpackage is that we have it already, thousands of packages uses it, and it works well and is fully capable of producing the source files consumed by dpkg-buildpackage etc to build the package. > > > One possible rebuttal to this is "gbp needs to do the right thing then". > > > Currently gbp by default generates a broken tarball, which is also a > > > source of confusion for many. > > > > Do you have a bug report number? > > No. > I've found #902249 which is titled "Pull tarballs from the archive (or > upstream location)", maybe that's the one you think about. I haven't read > it, except for the "I hoped we could stay out of that business in 2018 but > since tarballs are still _the_ _thing_ in Debian" line, which I liked. > > For the avoidance of doubt, I don't think gbp *can* do the right thing > there. It's not my rebuttal. Maybe gbp should just refuse to generate a > random tarball from a commit-ish and let^Wrequire people to provide one or > provide a way to generate one in a correct way. I often hear this complaint about pristine-tar, but I don't take it seriously until somebody actually files a bug report and has a reproducible case. For every single package I maintain I use pristine-tar and it works correctly. Personally I try to leverage upstream signatures both for tarballs and git tags as always when available, and for tarballs it requires pristine-tar, so I would not stop using pristine-tar as long as it works.
Re: Simpler git workflow for packaging with upstreamless repositories
Quoting Marco d'Itri (2024-11-26 23:34:24) > On Nov 26, Jonathan Dowland wrote: > > On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: > > > Yes, as they don't enable pristine-tar > > Is pristine-tar still valuable these days? > Not really. The debian branches of all of my packages[1] are based off > the upstream git repository, which is what I also use to review new > upstream releases and generate the .orig.tar.xz file for the archive. > I could not care less if that file is not bit per bit identical to the > upstream one (the git trees have signatures), or even if it does not > contain the same files (this is usually better, so I do not need to carry > around files which we like to rebuild anyway). If the upstream tarball is in the archive, it's probably okay to retrieve one from there. It's not always trivial because now you need to have the right "deb-src" lines enabled in your apt sources.list but it's possible. But how do you collaborate with others on packages that were not uploaded to the desired suite yet? Do you not use git for collaboration until the package has passed NEW? Thanks! cheers, josch signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Nov 26, Jonathan Dowland wrote: > On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: > > Yes, as they don't enable pristine-tar > Is pristine-tar still valuable these days? Not really. The debian branches of all of my packages[1] are based off the upstream git repository, which is what I also use to review new upstream releases and generate the .orig.tar.xz file for the archive. I could not care less if that file is not bit per bit identical to the upstream one (the git trees have signatures), or even if it does not contain the same files (this is usually better, so I do not need to carry around files which we like to rebuild anyway). Actually I use gbp.conf to make sure that pristine-tar is disabled. [1] except than the openbsd-derived ones, because CVS. -- ciao, Marco signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 09:25:12PM +0100, Simon Josefsson wrote: > Colin Watson writes: > > CI for a new-upstream-release commit you've pushed but haven't uploaded > > to the Debian archive yet, because you're waiting for CI. > > > > pristine-tar certainly isn't the only way here (it's true that CI could > > in principle fetch it directly from upstream), but I find it much easier > > than the alternatives. > > I often wait for CI before making an upload of a new upstream release, > even without pristine-tar, and Salsa CI hasn't had trouble automatically > finding a orig.tar.gz to work with for me. So maybe there is some > additional assumption for your situation? Maybe it runs uscan or something in that case; I confess I haven't checked. Still, as long as it's building source packages as part of the process, conceptually I prefer it having all the information in the repository rather than having to go out to an upstream site that might be unreliable. -- Colin Watson (he/him) [cjwat...@debian.org]
Re: Simpler git workflow for packaging with upstreamless repositories
Colin Watson writes: > On Tue, Nov 26, 2024 at 08:49:44PM +0100, Simon Josefsson wrote: >> If you haven't made an upload, then wouldn't you have the tarball >> locally while working on preparing the upload? >> >> And if someone doesn't have the orig.tar.gz locally, then why would >> anyone want to get it from a random git repository, rather than fetching >> it from the Debian archive or from upstream's release page? What is the >> use-case here that am I missing? > > CI for a new-upstream-release commit you've pushed but haven't uploaded > to the Debian archive yet, because you're waiting for CI. > > pristine-tar certainly isn't the only way here (it's true that CI could > in principle fetch it directly from upstream), but I find it much easier > than the alternatives. I often wait for CI before making an upload of a new upstream release, even without pristine-tar, and Salsa CI hasn't had trouble automatically finding a orig.tar.gz to work with for me. So maybe there is some additional assumption for your situation? /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 08:49:44PM +0100, Simon Josefsson wrote: > If you haven't made an upload, then wouldn't you have the tarball > locally while working on preparing the upload? > > And if someone doesn't have the orig.tar.gz locally, then why would > anyone want to get it from a random git repository, rather than fetching > it from the Debian archive or from upstream's release page? What is the > use-case here that am I missing? CI for a new-upstream-release commit you've pushed but haven't uploaded to the Debian archive yet, because you're waiting for CI. pristine-tar certainly isn't the only way here (it's true that CI could in principle fetch it directly from upstream), but I find it much easier than the alternatives. -- Colin Watson (he/him) [cjwat...@debian.org]
Re: Simpler git workflow for packaging with upstreamless repositories
Andrey Rakhmatullin writes: > On Tue, Nov 26, 2024 at 06:54:18PM +0100, Chris Hofstaedtler wrote: >> > >> > Yes, as they don't enable pristine-tar >> > >> >> > >> Is pristine-tar still valuable these days? >> > > >> > > Unfortunately yes. AFAIK the two options for fixing this that are >> > > usually proposed are: >> > > >> > > 1) treat it as a problem of each individual developer, just like >> > > pristine-tar. Instead of pristine-tar, invent new tooling to manage >> > > tarballs. >> > > This path often tries to solve the problem only for Debian and only >> > > in a narrow scenario. >> > > >> > > 2) Have all uploads always supply a new orig.tar.gz. This could mean >> > > either treating every package as Debian-native, or some other >> > > solution. >> > > This is a global solution and reduces complexity instead of adding >> > > to it. >> > >> > Until we record expected upstream tarball hashes in a debian/* file, an >> > acceptable approach seems to be to skip the pristine-tar branch and be >> > sure to download the previous orig.tar.* + orig.tar.*.asc from the >> > Debian archive, instead of attempting to re-generate it from the >> > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). >> >> This is 1). It cannot be done generically as it requires knowing >> where to download from, etc. > > The archive, when the tarball is already there. > > These suggestions never discuss what to do when the tarball was never > uploaded yet, even I didn't discuss that for simplicity. It makes sense > from some PoVs, at least when one doesn't use pristine-tar to make a > tarball that has differences in the actual content, not just bit > differences in the tarball itself while have identical file contents. If you haven't made an upload, then wouldn't you have the tarball locally while working on preparing the upload? And if someone doesn't have the orig.tar.gz locally, then why would anyone want to get it from a random git repository, rather than fetching it from the Debian archive or from upstream's release page? What is the use-case here that am I missing? I've always preferred to work with a pristine-tar branch myself, but I'm having trouble coming up with a strong motivation for its existance, so maybe backing down from that preference is a way forward. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 06:53:01PM +0100, Mechtilde Stehmann wrote: > > One possible rebuttal to this is "gbp needs to do the right thing then". > > Currently gbp by default generates a broken tarball, which is also a > > source of confusion for many. > > Do you have a bug report number? No. I've found #902249 which is titled "Pull tarballs from the archive (or upstream location)", maybe that's the one you think about. I haven't read it, except for the "I hoped we could stay out of that business in 2018 but since tarballs are still _the_ _thing_ in Debian" line, which I liked. For the avoidance of doubt, I don't think gbp *can* do the right thing there. It's not my rebuttal. Maybe gbp should just refuse to generate a random tarball from a commit-ish and let^Wrequire people to provide one or provide a way to generate one in a correct way. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 06:54:18PM +0100, Chris Hofstaedtler wrote: > > >> > Yes, as they don't enable pristine-tar > > >> > > >> Is pristine-tar still valuable these days? > > > > > > Unfortunately yes. AFAIK the two options for fixing this that are > > > usually proposed are: > > > > > > 1) treat it as a problem of each individual developer, just like > > > pristine-tar. Instead of pristine-tar, invent new tooling to manage > > > tarballs. > > > This path often tries to solve the problem only for Debian and only > > > in a narrow scenario. > > > > > > 2) Have all uploads always supply a new orig.tar.gz. This could mean > > > either treating every package as Debian-native, or some other > > > solution. > > > This is a global solution and reduces complexity instead of adding > > > to it. > > > > Until we record expected upstream tarball hashes in a debian/* file, an > > acceptable approach seems to be to skip the pristine-tar branch and be > > sure to download the previous orig.tar.* + orig.tar.*.asc from the > > Debian archive, instead of attempting to re-generate it from the > > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). > > This is 1). It cannot be done generically as it requires knowing > where to download from, etc. The archive, when the tarball is already there. These suggestions never discuss what to do when the tarball was never uploaded yet, even I didn't discuss that for simplicity. It makes sense from some PoVs, at least when one doesn't use pristine-tar to make a tarball that has differences in the actual content, not just bit differences in the tarball itself while have identical file contents. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
* Simon Josefsson [241126 16:27]: > Chris Hofstaedtler writes: > > > * Jonathan Dowland [241126 12:59]: > >> On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: > >> > Yes, as they don't enable pristine-tar > >> > >> Is pristine-tar still valuable these days? > > > > Unfortunately yes. AFAIK the two options for fixing this that are > > usually proposed are: > > > > 1) treat it as a problem of each individual developer, just like > > pristine-tar. Instead of pristine-tar, invent new tooling to manage > > tarballs. > > This path often tries to solve the problem only for Debian and only > > in a narrow scenario. > > > > 2) Have all uploads always supply a new orig.tar.gz. This could mean > > either treating every package as Debian-native, or some other > > solution. > > This is a global solution and reduces complexity instead of adding > > to it. > > Until we record expected upstream tarball hashes in a debian/* file, an > acceptable approach seems to be to skip the pristine-tar branch and be > sure to download the previous orig.tar.* + orig.tar.*.asc from the > Debian archive, instead of attempting to re-generate it from the > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). This is 1). It cannot be done generically as it requires knowing where to download from, etc. > I have never understood what value there is in duplicating the uploaded > tarball in the git repository. Recording a hash of it is sufficient. The hash is sufficient for knowing it changed, but you still have to get the actual tarball from somewhere. Chris
Re: Simpler git workflow for packaging with upstreamless repositories
Hello Andrey, Am 26.11.24 um 17:44 schrieb Andrey Rakhmatullin: On Tue, Nov 26, 2024 at 04:27:37PM +0100, Simon Josefsson wrote: One possible rebuttal to this is "gbp needs to do the right thing then". Currently gbp by default generates a broken tarball, which is also a source of confusion for many. Do you have a bug report number? -- Mechtilde Stehmann ## Debian Developer ## PGP encryption welcome ## F0E3 7F3D C87A 4998 2899 39E7 F287 7BBA 141A AD7F OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 09:44:31PM +0500, Andrey Rakhmatullin wrote: > Currently gbp by default generates a broken tarball, which is also a > source of confusion for many. I'd dare to even call this a bug. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Hope isn't a plan, but it's a hell of a drug. signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 04:27:37PM +0100, Simon Josefsson wrote: > >> > Yes, as they don't enable pristine-tar > >> > >> Is pristine-tar still valuable these days? > > > > Unfortunately yes. AFAIK the two options for fixing this that are > > usually proposed are: > > > > 1) treat it as a problem of each individual developer, just like > > pristine-tar. Instead of pristine-tar, invent new tooling to manage > > tarballs. > > This path often tries to solve the problem only for Debian and only > > in a narrow scenario. > > > > 2) Have all uploads always supply a new orig.tar.gz. This could mean > > either treating every package as Debian-native, or some other > > solution. > > This is a global solution and reduces complexity instead of adding > > to it. > > Until we record expected upstream tarball hashes in a debian/* file, an > acceptable approach seems to be to skip the pristine-tar branch and be > sure to download the previous orig.tar.* + orig.tar.*.asc from the > Debian archive, instead of attempting to re-generate it from the > upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). > > I have never understood what value there is in duplicating the uploaded > tarball in the git repository. Recording a hash of it is sufficient. > But I'm also happy to work with pristine-tar branches when that is the > workflow for a particular package. I just wish the tooling handled > *.asc files better, and stored them too automatically. One possible rebuttal to this is "gbp needs to do the right thing then". Currently gbp by default generates a broken tarball, which is also a source of confusion for many. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Chris Hofstaedtler writes: > * Jonathan Dowland [241126 12:59]: >> On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: >> > Yes, as they don't enable pristine-tar >> >> Is pristine-tar still valuable these days? > > Unfortunately yes. AFAIK the two options for fixing this that are > usually proposed are: > > 1) treat it as a problem of each individual developer, just like > pristine-tar. Instead of pristine-tar, invent new tooling to manage > tarballs. > This path often tries to solve the problem only for Debian and only > in a narrow scenario. > > 2) Have all uploads always supply a new orig.tar.gz. This could mean > either treating every package as Debian-native, or some other > solution. > This is a global solution and reduces complexity instead of adding > to it. Until we record expected upstream tarball hashes in a debian/* file, an acceptable approach seems to be to skip the pristine-tar branch and be sure to download the previous orig.tar.* + orig.tar.*.asc from the Debian archive, instead of attempting to re-generate it from the upstream/ branch (which isn't guaranteed to be bit-by-bit reproducible). I have never understood what value there is in duplicating the uploaded tarball in the git repository. Recording a hash of it is sufficient. But I'm also happy to work with pristine-tar branches when that is the workflow for a particular package. I just wish the tooling handled *.asc files better, and stored them too automatically. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 11:59:26AM +, Jonathan Dowland wrote: > > Yes, as they don't enable pristine-tar > > Is pristine-tar still valuable these days? Extremely, unless your workflow requires using something else to get the upstream tarball from the archive (among other less important requirements). Though many people don't use d/gbp.conf for this, with predictable results when someone else than those people tries to build the repo. > > and cannot guess your branch naming system. > > I haven't checked, but in an ideal world gbp would default to the same > values for these as described in DEP-14. In that same ideal world, we'd > have DEP-14 out of Draft status. Yes, and all the teams would use it in that ideal world. There is a bug reported against gbp for the first of these points AFAIK. That would be a backwards incompatible change though I guess. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
* Jonathan Dowland [241126 12:59]: > On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: > > Yes, as they don't enable pristine-tar > > Is pristine-tar still valuable these days? Unfortunately yes. AFAIK the two options for fixing this that are usually proposed are: 1) treat it as a problem of each individual developer, just like pristine-tar. Instead of pristine-tar, invent new tooling to manage tarballs. This path often tries to solve the problem only for Debian and only in a narrow scenario. 2) Have all uploads always supply a new orig.tar.gz. This could mean either treating every package as Debian-native, or some other solution. This is a global solution and reduces complexity instead of adding to it. > > and cannot guess your branch naming system. > > I haven't checked, but in an ideal world gbp would default to the same > values for these as described in DEP-14. In that same ideal world, we'd > have DEP-14 out of Draft status. IIRC gbp has an open bug for that in the state of "please send patches that do not break backwards compat". DEP14 also IMO needs to make a call on the layout, and not leave options, for any tooling to support it. Chris
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue Nov 26, 2024 at 10:50 AM GMT, Andrey Rakhmatullin wrote: Yes, as they don't enable pristine-tar Is pristine-tar still valuable these days? and cannot guess your branch naming system. I haven't checked, but in an ideal world gbp would default to the same values for these as described in DEP-14. In that same ideal world, we'd have DEP-14 out of Draft status. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Simpler git workflow for packaging with upstreamless repositories
On Tue, Nov 26, 2024 at 08:51:43AM +, Jonathan Dowland wrote: > How common debian/gbp.conf points at something else: perhaps gbp's defaults > are not good, if that many packages need to override them. Yes, as they don't enable pristine-tar and cannot guess your branch naming system. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Fri Nov 22, 2024 at 11:29 AM GMT, Simon Josefsson wrote: I thought about the gbp aspect of the lets-move-things-to-salsa and I suggest to consider if they are better left orthogonal. Could you make one DEP for lets-move-to-salsa and one DEP for lets-document-a-build-workflow? I haven't read Otto's rewrite of DEP-18 yet, but that was my feeling from the first iteration: it's a really grand project trying to do lots at once. I'd like to see some progress in this space and I think the chance of success is much better with smaller, more targeted proposals, that can pass muster and be adopted in series. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu Nov 21, 2024 at 4:10 AM GMT, Otto Kekäläinen wrote: There is a debian/gbp.conf in 13573 packages in Debian This is an interesting proxy measure for the prevalence of gbp usage. It probably undercounts: I sometimes use gbp but I've only got this file in one of my sources (one I recently inherited from others). How common debian/gbp.conf points at something else: perhaps gbp's defaults are not good, if that many packages need to override them. -- Please do not CC me for listmail. 👱🏻 Jonathan Dowland ✎j...@debian.org 🔗 https://jmtd.net
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, Quoting Otto Kekäläinen (2024-11-23 00:09:41) > You can have upstream git and tarballs at the same time, and even have DFSG > cleanup take place and git show you exactly the differences of all the > versions. I'm interested in the DFSG cleanup. How can this be done while having an upstream git repo that ships these files? Your example of src:entr does not seem to have any Files-Excluded. > If you look at > https://salsa.debian.org/debian/entr/-/network/debian%2Flatest?extended_sha1=debian%2Flatest&filter_ref=1 > (or better, gbp clone it locally to more easily browse it with `gitk --all`) > you can see how the upstream release git tag "5.6" was merged on branch > 'upstream/latest' which is the target of tarball imports, and that was then > merged on the 'debian/latest' branch. Commands how to do this are in > https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/README.source.md This is a really good write-up. Can we have this in a more prominent place than in a README.source of some "random" package? > and the > https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/gbp.conf has > the configs so git-buildpackage can be used without the need to constantly > pass it information about upstream git tag format etc. That gbp.conf touches 15 settings. Can gbp not do the right thing by default? > For dfsg-filtering the watchfile options only apply for uscan/tarball. To > ensure the upstream/latest branch stays dfsg-clean on git merges, configure > 'filter' in debian/gbp.conf. You mean in [import-orig]? Will this need to duplicate the entries that already exist in Files-Excluded? > Which package do you have DFSG tarballs? I can take a look and help you > convert it into something where the import is as automatic as possible. The import is automatic. I just run "uscan --verbose" and then "gbp import orig" and uscan will take care of using Files-Excluded to clean up the tarball. Thanks! cheers, josch signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi! > > I am contemplating if I should also make videos on Debian packaging best > > practices, or have start a new Matrix channel specifically to help > > maintainers setup their git repositories correctly and find the optimal > > git-buildpackage commands for each thing they want to do. > > I'd describe myself of being in the camp of "I don't care about the commands, i > don't care how my git branches are named, just document one thing so that my > packages can do the same thing that most other packages do". For that reason I > so far created new package repos with "gbp import-dsc" and used all the > defaults that come with that. I don't think I care much for what these defaults > are at least I didn't see myself reading past discussions about this and > thinking "no don't name this branch debian/latest instead of debian/foo" or > something. I like to read of other people's workflows but then I often do not > see how their workflows can possibly fit my packages. There seem to be many > people who have the upstream git as part of their packaging git. I'm happy that > works for them but I don't see how I can leave my tarball-centered workflows > (even though all my upstream work in git) if all my upstreams ship DFSG > non-free material which I have to remove from their tarballs first. You can have upstream git and tarballs at the same time, and even have DFSG cleanup take place and git show you exactly the differences of all the versions. If you look at https://salsa.debian.org/debian/entr/-/network/debian%2Flatest?extended_sha1=debian%2Flatest&filter_ref=1 (or better, gbp clone it locally to more easily browse it with `gitk --all`) you can see how the upstream release git tag "5.6" was merged on branch 'upstream/latest' which is the target of tarball imports, and that was then merged on the 'debian/latest' branch. Commands how to do this are in https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/README.source.md and the https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/gbp.conf has the configs so git-buildpackage can be used without the need to constantly pass it information about upstream git tag format etc. For dfsg-filtering the watchfile options only apply for uscan/tarball. To ensure the upstream/latest branch stays dfsg-clean on git merges, configure 'filter' in debian/gbp.conf. Which package do you have DFSG tarballs? I can take a look and help you convert it into something where the import is as automatic as possible.
Re: Simpler git workflow for packaging with upstreamless repositories
Quoting Simon Josefsson (2024-11-22 12:54:02) > Doesn't 'gbp import-orig --uscan' work if you have a watch-file like this: > > version=4 > opts="mode=git, pgpmode=none,\ > dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" \ > https://github.com/withfig/autocomplete-tools.git \ > HEAD debian but this uses mode=git. The documentation in uscan(1) very clearly says about the git mode: > If the upstream publishes the released tarball via its web interface, please > use it instead of using this mode. This mode is the last resort method. Or maybe this part of the documentation is just not accurate anymore and things have changed? > Also is there a problem to mirror upstream's non-DFSG content on Salsa? > > The important aspect seems to be that *.orig.tar.* and debian/* uploaded > into the archive shouldn't contain non-DFSG stuff if it targets main. I > thought Salsa can be used to maintain contrib/non-free/non-free-firmware > projects too, so I assume there is no restriction that Salsa git repositories > may only contain DFSG-content. Should our users (and that includes people who want to change the source) use the orig.tar and dsc for their development or should they not use the packaging git instead if they want to make changes or contribute? If they use the latter, should what they are using there not adhere to the principles of the DFSG? If I run "apt-get source tinyusb" then it correctly suggests: > NOTICE: 'tinyusb' packaging is maintained in the 'Git' version control system > at: > https://salsa.debian.org/debian/tinyusb.git > Please use: > git clone https://salsa.debian.org/debian/tinyusb.git > to retrieve the latest (possibly unreleased) updates to the package. I don't think it would be nice if retrieving that git repo (and all the branches needed for working with the package) would require downloading non-free content even though tinyusb is in main. I don't think this is written down anywhere, so maybe I'm just making life harder for myself and maybe instead I should not feel so bad about using salsa as a hosting platform for non-free content and then distributing that non-free content to users who want to hack on Debian "main"? Thanks! cheers, josch signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
Johannes Schauer Marin Rodrigues writes: > Quoting Simon Josefsson (2024-11-22 12:18:36) >> > I like to read of other people's workflows but then I often do not see how >> > their workflows can possibly fit my packages. There seem to be many people >> > who have the upstream git as part of their packaging git. I'm happy that >> > works for them but I don't see how I can leave my tarball-centered >> > workflows (even though all my upstream work in git) if all my upstreams >> > ship DFSG non-free material which I have to remove from their tarballs >> > first. >> This works fairly well these days: use Files-Excluded: in d/copyright and >> "dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" in d/watch. Tools like gbp, >> uscan, origtargz seems to do the right thing automatically. > > That's what I'm doing. But that works with tarballs not with upstream > as git. If upstream (deliberately, so this will not change) has > DSFG-non-free content in it, then I should not copy that into a git > packaging repo that is targeting main. Removing the problematic parts > from the upstream git repos would rewrite their history, invalidate > tags etc, so the result would not be very useful anymore. Thus, I > usually have one directory on my PC with the upstream git and another > with my Debian packaging git. The packaging git does not have the > upstream git with non-free content int it. Instead my packaging git > regularly imports new upstream releases as tarballs and removes the > non-free content via Files-Excluded and dversionmangle/repacksuffix in > d/watch. I don't see how I can get rid of the tarballs here but > instead embrace the (non-free) upstream git. Maybe I'm missing > something? Doesn't 'gbp import-orig --uscan' work if you have a watch-file like this: version=4 opts="mode=git, pgpmode=none,\ dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" \ https://github.com/withfig/autocomplete-tools.git \ HEAD debian Also is there a problem to mirror upstream's non-DFSG content on Salsa? The important aspect seems to be that *.orig.tar.* and debian/* uploaded into the archive shouldn't contain non-DFSG stuff if it targets main. I thought Salsa can be used to maintain contrib/non-free/non-free-firmware projects too, so I assume there is no restriction that Salsa git repositories may only contain DFSG-content. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Quoting Simon Josefsson (2024-11-22 12:18:36) > > I like to read of other people's workflows but then I often do not see how > > their workflows can possibly fit my packages. There seem to be many people > > who have the upstream git as part of their packaging git. I'm happy that > > works for them but I don't see how I can leave my tarball-centered > > workflows (even though all my upstream work in git) if all my upstreams > > ship DFSG non-free material which I have to remove from their tarballs > > first. > This works fairly well these days: use Files-Excluded: in d/copyright and > "dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" in d/watch. Tools like gbp, > uscan, origtargz seems to do the right thing automatically. That's what I'm doing. But that works with tarballs not with upstream as git. If upstream (deliberately, so this will not change) has DSFG-non-free content in it, then I should not copy that into a git packaging repo that is targeting main. Removing the problematic parts from the upstream git repos would rewrite their history, invalidate tags etc, so the result would not be very useful anymore. Thus, I usually have one directory on my PC with the upstream git and another with my Debian packaging git. The packaging git does not have the upstream git with non-free content int it. Instead my packaging git regularly imports new upstream releases as tarballs and removes the non-free content via Files-Excluded and dversionmangle/repacksuffix in d/watch. I don't see how I can get rid of the tarballs here but instead embrace the (non-free) upstream git. Maybe I'm missing something? Thanks! cheers, josch signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
Otto Kekäläinen writes: >> in addition to the above: gbp is great if it works, if it doesn't I can >> start to learn to debug a complex system, while on the contrary, if I use >> git and debuild/pbuilder/sbuild manually all the time, I know those tools, >> they are rather easy to debug etc. > > Personally I think gbp is very easy to debug. You just add --verbose > to any command and it will print out what it is doing. Example: > > ± gbp import-orig --uscan --verbose ... I thought about the gbp aspect of the lets-move-things-to-salsa and I suggest to consider if they are better left orthogonal. Could you make one DEP for lets-move-to-salsa and one DEP for lets-document-a-build-workflow? The former would not talk a bout local builds at all. The latter would not talk about the Salsa workflow at all. They both intersect on git branch naming, and maybe some other minor aspects, but don't we have that already in DEP 14? Compare making packaging contributions to Debian to packaging contributions to Homebrew, Alpine, OpenWRT, ArchLinux, Guix, etc. There is a possibility to reach a completely git-based Salsa workflow for Debian packages that doesn't involve any local builds on people's laptops at all. This is comparable to Homebrew contributions, I find them a joy to work with in comparison and can contribute even without having any Homebrew development environment setup, or even without physical access to a Mac. There is no reason Debian packaging contributions couldn't work like that. Yes, old timers will continue build stuff on their laptop, and that has to be fine. We need to resolve how to authenticate archive uploads chaining back to a DD instead of PGP on tarball, but that is doable. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Johannes Schauer Marin Rodrigues writes: > I like to read of other people's workflows but then I often do not see > how their workflows can possibly fit my packages. There seem to be > many people who have the upstream git as part of their packaging > git. I'm happy that works for them but I don't see how I can leave my > tarball-centered workflows (even though all my upstream work in git) > if all my upstreams ship DFSG non-free material which I have to remove > from their tarballs first. This works fairly well these days: use Files-Excluded: in d/copyright and "dversionmangle=s/\+ds\d*$//,repacksuffix=+ds" in d/watch. Tools like gbp, uscan, origtargz seems to do the right thing automatically. > So given all this, the point I wanted to make was: I'd like to watch your > videos if you make them. But at least for me you do not have to go through the > trouble of shooting videos. Just some HTML docs would be just fine for me. > Currently, I'm missing a long-term maintained document which explains "the" > (haha) recommended Debian git workflow through the lifetime of a package from > its initial creation to backports or stable updates. Yes please! There are so many details (library transitions? non-dfsg files? experimental->sid migration? go reverse-rebuilds? etc) that are not well documented. I have my own big README with various workflow commands but I deal with packages following perhaps 10+ different styles today, and I try to respect the traditional style used in a package. /Simon signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On 18/11/2024 18:16, Kari Pahula wrote: > I'm thinking that it's the merging of debian and upstream branches and > maintaining it that really causes the warts in gbp and I'm not at all > sure if there are any actual workflows that require having that. > "upstreamless" as I described it may be a bit too much for general use > but could there be a case for doing everything with a mergeless model > instead? Could we turn the current packaging format: foo-1.2.3/ -> upstream source src/ ... debian/ -> Debian inserted things where debian/ must reside *inside* the upstream source; inside out: foo-1.2.3-debian/ ... Debian things upstream/ -> upstream source src/ ... where "upstream source" could be extracted from a tarball, a cloned Git repository checked out at a specified commit, a Git submodule, etc. as the maintainer pleases. However it's produced, it is now a "guest" of the packaging (which now becomes the "host"), allowing the host to do more, unlike the current way around, where the packaging is the guest and pretty limited. So we no longer have the problem of "merging of debian and upstream branches"? But as wRAR stated in their other email (though not directly related): > Yes, changing Debian so that the required workflows are more simple is > much better but also impossible. This whimsical "4.0 (inside-out)" format is only for your entertainment. :> -- Sdrager, Blair Noctis OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, > > and you seem to think adding another Debian specific complex tool will > > attrack new people? Most people^wcontributors out there know git already, > > for them a simple git only workflow is *much* more inviting than having > > to learn gbp *only* to contribute to Debian. > > This. Exactly this. > > This is the reason I can contribute small improvements to Alpine and > other projects with reasonable effort for small improvements. The > tools are the same tools I already use elsewhere, and stuff is as > transparent as it needs to be for outside contributors. > > Time to get build people where they are, and not where we are. As Andrey wrote, sure this would be better, but also impossible. How .dsc/.deb/.orig.tar.gz and occasional modified .tar.gz work in Debian is fundamentally very different from how in e.g. Alpine a single APKBUILD defines a the upstream source a just an URL and a sha512 to verify the download was correct. Maybe one day in the future we change the Debian source format to not have upstream tarball bundled, but that is a totally different discussion from what we are doing now in DEP-18: recognizing the current most common workflows and elevating them as the recommended baseline for those who are keen to collaborate more efficiently. In Debian/DEP-18 for all the normal daily packaging work like pulling and pushing, making commits, amending commits, creating branches, rebasing branches etc you can and should use just regular git commits. Git-buildpackage is there only to tell people that is the tool to interact with repos. Git-buildpackage does what it does out of necessity to be able to produce the *.changes and *.dsc files. Therefore the upstream import and building packages need to be done in a certain way and there are 2 or 3 branches in the git repositories. Luckily, these 3 branch names are unified to be debian/latest, upstream/latest and pristine-tar thanks to DEP-14. Now with DEP-18 we can move along and make it easier for new people and also tenured maintainers to land on the same best practices. Going back to the example of Alpine, they enforce heavily one single workflow that includes one single VCS (git), one single hosting place (https://gitlab.alpinelinux.org/) and to my knowledge also one single tool to build and update packages. DEP-18 does not enforce anything, but if you want to have a situation that people voluntarily move towards adopting the same base workflow for undifferentiated Debian packaging, you should support DEP-18 and not dismiss it.
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, > > > fwiw, I'm also not using git-buildpackage and I don't want to. (Step > > > learning > > > curve, too much magic, hard to debug and I dont see benefits for the > > > packages > > > I maintain.) I'm also not a beginner. And I love gbp-dch. > > I think git-buildpackage is great, and I am a happy user. However, it > > took me quite a while to figure out what is the optimal way to use it, > > as it has so many options. Can you elaborate what is the specific > > action/workflow you think git-buildpackage you were frustrated with? > > sigh. see above. I obviously saw above and I think asking for more details I think is fair so we can discuss if the thing you experienced as "hard to learn" or "had too much magic" was fundamentally bad and can't be fixed, or perhaps if better docs or better error handling etc can make that frustration go away. I feel we should try to polish and improve the existing tooling that the majority uses, rather than giving up on it and demand an entirely new tool or workflow. > in addition to the above: gbp is great if it works, if it doesn't I can > start to learn to debug a complex system, while on the contrary, if I use > git and debuild/pbuilder/sbuild manually all the time, I know those tools, > they are rather easy to debug etc. Personally I think gbp is very easy to debug. You just add --verbose to any command and it will print out what it is doing. Example: ± gbp import-orig --uscan --verbose gbp:debug: ['git', 'rev-parse', '--show-cdup'] gbp:debug: ['git', 'rev-parse', '--is-bare-repository'] gbp:debug: ['git', 'rev-parse', '--git-dir'] gbp:debug: ['git', 'for-each-ref', '--format=%(refname:short)', 'refs/heads/'] gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream'] gbp:error: Repository does not have branch 'upstream' for upstream sources. If there is none see file:///usr/share/doc/git-buildpackage/manual-html/gbp.import.html#GBP.IMPORT.CONVERT on howto create it otherwise use --upstream-branch to specify it. I am sure you already knew this, but pasting here an example for the sake of discussion. The output and debuggability can for sure further be improved still, I am not saying gbp is perfect, but it is by far the best we have and deserves to be the thing any team or large group standardize on today to have an easier time collaborating. > > I think this will be more successful if you frame this as good patterns > > to follow for people who wish to opt-in on the premise of adopting this > > workflow. That framing follows other DEP's. Now it reads as a mandate > > for everything. Acknowledging existance of exceptions to the rules > > within the rules may also help to gain acceptance. > > This! Thanks for the feedback, I will try to further iterate on the introduction part in DEP-18 to make is more clear what is the purpose and that a DEP-18 is not a policy and is opt-in like all the other DEPs.
Re: Simpler git workflow for packaging with upstreamless repositories
On Wednesday, November 20, 2024 9:10:30 PM MST Otto Kekäläinen wrote: > I am not enforcing git-buildpackage on everyone. I am accelerating the > convergence of workflows so that it is easier for maintainers to > collaborate, so we can have more code reviews, more new maintainers, > and in the long run perhaps decrease the number of single-maintainer > packages for packages where the maintainer does not want to be the > sole maintainer. Currently the learning curve on how to contribute > prevents many people from doing it. I am all for this. I don’t know if we will ever get the number of Debian workflows down to 1, but I would like to see it somewhere lower than 1,000 (that is only a slight exaggeration) in my lifetime. -- Soren Stoutner so...@debian.org signature.asc Description: This is a digitally signed message part.
Re: Simpler git workflow for packaging with upstreamless repositories
Hi, Quoting Otto Kekäläinen (2024-11-19 07:34:45) > I am contemplating if I should also make videos on Debian packaging best > practices, or have start a new Matrix channel specifically to help > maintainers setup their git repositories correctly and find the optimal > git-buildpackage commands for each thing they want to do. I'd describe myself of being in the camp of "I don't care about the commands, i don't care how my git branches are named, just document one thing so that my packages can do the same thing that most other packages do". For that reason I so far created new package repos with "gbp import-dsc" and used all the defaults that come with that. I don't think I care much for what these defaults are at least I didn't see myself reading past discussions about this and thinking "no don't name this branch debian/latest instead of debian/foo" or something. I like to read of other people's workflows but then I often do not see how their workflows can possibly fit my packages. There seem to be many people who have the upstream git as part of their packaging git. I'm happy that works for them but I don't see how I can leave my tarball-centered workflows (even though all my upstream work in git) if all my upstreams ship DFSG non-free material which I have to remove from their tarballs first. So given all this, the point I wanted to make was: I'd like to watch your videos if you make them. But at least for me you do not have to go through the trouble of shooting videos. Just some HTML docs would be just fine for me. Currently, I'm missing a long-term maintained document which explains "the" (haha) recommended Debian git workflow through the lifetime of a package from its initial creation to backports or stable updates. Thank you for your work! cheers, josch signature.asc Description: signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Thu, Nov 21, 2024 at 11:26:15AM +, Holger Levsen wrote: > > I am not enforcing git-buildpackage on everyone. I am accelerating the > > convergence of workflows so that it is easier for maintainers to > > collaborate, so we can have more code reviews, more new maintainers, > > and in the long run perhaps decrease the number of single-maintainer > > packages for packages where the maintainer does not want to be the > > sole maintainer. > > and you seem to think adding another Debian specific complex tool will > attrack new people? Most people^wcontributors out there know git already, > for them a simple git only workflow is *much* more inviting than having > to learn gbp *only* to contribute to Debian. gbp is useless outside > Debian (and derivatives. Yet, thats only a part of the free software world.) Yes, changing Debian so that the required workflows are more simple is much better but also impossible. -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi Kari (2024.11.18_15:40:55_+) > - Only store debian/ in the repository and use origtargz for the rest. I used to feel strongly this way, too. However, A big advantage of storing the upstream sources in git is that you can use git to manage the quilt patch queue. I used to be pretty good at editing patches to get them to apply after upstream changed the code, but its painful and slow work. gbp-pq rebase is *infinitely* better. You need to be comfortable in git rebasing. But if you use git, you are probably already a git rebase ninja. You can use those same skills in Debian patch queue maintenance. I'd suggest giving gbp-pq a good try before writing off the gbp stack. Maintaining a complex patch stack is *much* easier with it. I look after packages in both layouts, and I am sold on storing upstream sources in git. Stefano -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Simpler git workflow for packaging with upstreamless repositories
* Holger Levsen [241121 12:26]: > and you seem to think adding another Debian specific complex tool will > attrack new people? Most people^wcontributors out there know git already, > for them a simple git only workflow is *much* more inviting than having > to learn gbp *only* to contribute to Debian. This. Exactly this. This is the reason I can contribute small improvements to Alpine and other projects with reasonable effort for small improvements. The tools are the same tools I already use elsewhere, and stuff is as transparent as it needs to be for outside contributors. Time to get build people where they are, and not where we are. Chris
Re: Simpler git workflow for packaging with upstreamless repositories
On Wed, Nov 20, 2024 at 08:10:30PM -0800, Otto Kekäläinen wrote: > > fwiw, I'm also not using git-buildpackage and I don't want to. (Step > > learning > > curve, too much magic, hard to debug and I dont see benefits for the > > packages > > I maintain.) I'm also not a beginner. And I love gbp-dch. > I think git-buildpackage is great, and I am a happy user. However, it > took me quite a while to figure out what is the optimal way to use it, > as it has so many options. Can you elaborate what is the specific > action/workflow you think git-buildpackage you were frustrated with? sigh. see above. in addition to the above: gbp is great if it works, if it doesn't I can start to learn to debug a complex system, while on the contrary, if I use git and debuild/pbuilder/sbuild manually all the time, I know those tools, they are rather easy to debug etc. > I am not enforcing git-buildpackage on everyone. I am accelerating the > convergence of workflows so that it is easier for maintainers to > collaborate, so we can have more code reviews, more new maintainers, > and in the long run perhaps decrease the number of single-maintainer > packages for packages where the maintainer does not want to be the > sole maintainer. and you seem to think adding another Debian specific complex tool will attrack new people? Most people^wcontributors out there know git already, for them a simple git only workflow is *much* more inviting than having to learn gbp *only* to contribute to Debian. gbp is useless outside Debian (and derivatives. Yet, thats only a part of the free software world.) On Thu, Nov 21, 2024 at 09:51:15AM +0100, Simon Josefsson wrote: > Otto Kekäläinen writes: > I think this will be more successful if you frame this as good patterns > to follow for people who wish to opt-in on the premise of adopting this > workflow. That framing follows other DEP's. Now it reads as a mandate > for everything. Acknowledging existance of exceptions to the rules > within the rules may also help to gain acceptance. This! ...and hopefully last post on this thread from me. I've said everything I wanted to say. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ The upcoming clima apocalypse is the big elephant in every room now. signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi! > fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning > curve, too much magic, hard to debug and I dont see benefits for the packages > I maintain.) I'm also not a beginner. And I love gbp-dch. I think git-buildpackage is great, and I am a happy user. However, it took me quite a while to figure out what is the optimal way to use it, as it has so many options. Can you elaborate what is the specific action/workflow you think git-buildpackage you were frustrated with? Perhaps I can help you discover the optimal workflow and you can skip the learning curve? > please dont enforce git-buildpackage on everyone. I am not enforcing git-buildpackage on everyone. I am accelerating the convergence of workflows so that it is easier for maintainers to collaborate, so we can have more code reviews, more new maintainers, and in the long run perhaps decrease the number of single-maintainer packages for packages where the maintainer does not want to be the sole maintainer. Currently the learning curve on how to contribute prevents many people from doing it. As part of this I help people make the most out of git-buildpackage, which is by far the most popular tool for this, and to my knowledge the only tool that supports having upstream branch in the same repository, cherry-picking upstream commits easily, rebasing Debian patches on upstream, tracking supply-chain from upstream git tags and tarballs (or both) to Debian releases and so forth. There is a debian/gbp.conf in 13573 packages in Debian, and probably twice as many in total use git-buildpackage already. Almost all team policies I read have converged on git-buildpackage. If we don't converge on what the majority is already using, what then? > emacs might be better than nano for many usecases, but not all. if someone > would force me to use emacs, I'd be out. (FWIW, i'm a vim user, this is an > analogy.) The analogy with an editor does not work, as the editor is your personal tool. A better analogy would be interactive pair-programming, and in such a setting would need to agree with your pair to use for example Pulsar or Zed so that there is a common system that shows on your screens what the other person is typing. If you had bad experiences with git-buildpackage, please share them with me, I can help you find the optimal workflow, and you can perhaps give git-buildpackage a second chance?
Re: Simpler git workflow for packaging with upstreamless repositories
Hello Kari, Have you seen: https://wiki.debian.org/GitPackagingSurvey You may find something suitable for what you want there. -- Sean Whitton
Re: Simpler git workflow for packaging with upstreamless repositories
hi, fwiw, I'm also not using git-buildpackage and I don't want to. (Step learning curve, too much magic, hard to debug and I dont see benefits for the packages I maintain.) I'm also not a beginner. And I love gbp-dch. just as another data point. (the above.) please dont enforce git-buildpackage on everyone. emacs might be better than nano for many usecases, but not all. if someone would force me to use emacs, I'd be out. (FWIW, i'm a vim user, this is an analogy.) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Segregation was legal. Slavery was legal. Don't use legality as a guide to morality. signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Hi! > With all the recent talk about increasing the use of git for > packaging, I'd like to address one thing: git-buildpackage is very > complex to use. As an alternative, I'd like to propose a simpler > setup: .. > sick of it and I can tell you it only makes me want to ignore you. If > you ask me it's git-buildpackage's irreducible difficulty of use > that's the largest road block to increase the use of git for > packaging. I think git-buildpackage is great, and I am a happy user. It did however take me quite a while to figure out what is the best practice to use it, as it has so many options. I see you are also using it at https://salsa.debian.org/science-team/chuffed so you seem to have learned to use it. Can you elaborate, is there a specific thing you think git-buildpackage falls short on? Instead of trying to write a new tool from scratch, I hope more people would offer help to Guido to maintain and polish git-buildpackage and docs. I am for example proposing doc improvements in https://salsa.debian.org/agx/git-buildpackage/-/merge_requests/6 and I tried make git-buildpackage use DEP-14 branches (debian/latest and upstream/latest) by default in https://salsa.debian.org/agx/git-buildpackage/-/merge_requests/1, although I ran out of steam with this latter one as it turned out to be quite complex. I was also planning to suggest updates the the Developers Reference on how to use git-buildpackage in the context of daily package maintenance tasks so people would more easily discover them (draft at https://salsa.debian.org/debian/developers-reference/-/merge_requests/63). To make it easier for contributors I publish in my packages debian/gbp.conf, so the settings are automatically correct, and also README.source(.md) so contributors can discover the correct gbp comands easily. See examples at https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/gbp.conf and https://salsa.debian.org/debian/entr/-/blob/debian/latest/debian/README.source.md. I am contemplating if I should also make videos on Debian packaging best practices, or have start a new Matrix channel specifically to help maintainers setup their git repositories correctly and find the optimal git-buildpackage commands for each thing they want to do.
Re: Simpler git workflow for packaging with upstreamless repositories
On Mon, Nov 18, 2024 at 05:40:55PM +0200, Kari Pahula wrote: > With all the recent talk about increasing the use of git for > packaging, I'd like to address one thing: git-buildpackage is very > complex to use. As an alternative, I'd like to propose a simpler > setup: > > - Only store debian/ in the repository and use origtargz for the rest. > > - One branch per distribution you target. > > - Only tag debian revisions. > > That's it, less is more. The master branch would be just a single > debian directory. What it specifically wouldn't have is an upstream > branch and no extracted upstream sources in any permanent branch. Let me start by saying that I understand where you're coming from; any tool that allows different use scenarios will almost inevitably grow more and more complex with time, and become more and more difficult to use by beginners, unless there are good tutorials and step-by-step recipes for the most common cases. TBH, I think that the git-buildpackage manual is relatively easy to read and understand, but, of course, that opinion of mine is tainted by the fact that I cannot exactly be called a beginner :) (although there are new things I learn about Debian packaging every month) However... different people are used to different workflows. I personally really, really like the fact that I have the Git history of all the upstream files in the same repo and I don't have to go over to a different repo to figure out what changed when. Also, I like that immediately after `gbp import-orig` I can run `git show upstream` to review the diff (TBH, me being very pedantic, I usually have already given it a quick glance using `diff -urN` on unpacked tarballs before even importing it, if only to figure out if there are new files that I need to exclude, but that's not always the case), and that I can at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path` or similar commands. I have even been known to use `git bisect` in a Debian package directory in some weird cases. And yes, all of that can be handled in a separate Git repo with a clean checkout of the upstream repository without any of the Debian fluff; however, that would require me switching between terminals/tmux panes/whatever, and sometimes that seems like too much work when I can have it all in a single repository :) So... yes, a simpler setup would work for some people, and it may be better for beginners. However, there are some benefits to a full repository containing both the upstream source and the Debian changes, and some people like to use them every now and then. Still, thanks for your desire to make working on Debian easier and better! G'luck, Peter -- Peter Pentchev r...@ringlet.net r...@debian.org pe...@morpheusly.com PGP key:https://www.ringlet.net/roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Mon, Nov 18, 2024 at 08:16:59PM +0200, Kari Pahula wrote: > > However... different people are used to different workflows. > > I personally really, really like the fact that I have the Git history of > > all the upstream files in the same repo and I don't have to go over to > > a different repo to figure out what changed when. Also, I like that > > immediately after `gbp import-orig` I can run `git show upstream` to > > review the diff (TBH, me being very pedantic, I usually have already > > given it a quick glance using `diff -urN` on unpacked tarballs before > > even importing it, if only to figure out if there are new files that > > I need to exclude, but that's not always the case), and that I can > > at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path` > > or similar commands. I have even been known to use `git bisect` in > > a Debian package directory in some weird cases. > > > > And yes, all of that can be handled in a separate Git repo with > > a clean checkout of the upstream repository without any of > > the Debian fluff; however, that would require me switching between > > terminals/tmux panes/whatever, and sometimes that seems like too much > > work when I can have it all in a single repository :) > > Preface, just in case: I list a few git commands, don't try running > them if you have uncommitted changes. > > Okay, I could amend what I proposed originally with this option: Do > the debian work in a fork of upstream's repository, but never merge > the debian branch and upstream branch. That is, the start of Debian > maintenance would be by cloning upstream and then with "git checkout > --orphan debian" followed by "git reset --hard". > > Do you think you could do all of the above with that model, with a > command like "git checkout debian -- ." to insert the debian contents > to an upstream branch? > > I'm thinking that it's the merging of debian and upstream branches and > maintaining it that really causes the warts in gbp and I'm not at all > sure if there are any actual workflows that require having that. > "upstreamless" as I described it may be a bit too much for general use > but could there be a case for doing everything with a mergeless model > instead? > > Furthermore, I think import-orig should really be only about > establishing a particular byte string as the orig.tar. Think of > object storage. If there's a connection to a particular commit hash > or release tag in the repository it would better be represented as a > separate entry. Perhaps as a text file like debian/upstream-commits > that would be a list of upstream version/commit hash or tag pairs. > From where I stand, the way these disparate concerns have been merged > seems to be one root cause for all sorts of unintended consequences. You are talking about the upstream git history while Peter is talking about simply importing orig tarballs. Maybe what is causing warts in gbp *for you* is some *specific* gbp workflow, one of the many it supports? -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Mon, Nov 18, 2024 at 06:19:58PM +0200, Peter Pentchev wrote: > However... different people are used to different workflows. > I personally really, really like the fact that I have the Git history of > all the upstream files in the same repo and I don't have to go over to > a different repo to figure out what changed when. Also, I like that > immediately after `gbp import-orig` I can run `git show upstream` to > review the diff (TBH, me being very pedantic, I usually have already > given it a quick glance using `diff -urN` on unpacked tarballs before > even importing it, if only to figure out if there are new files that > I need to exclude, but that's not always the case), and that I can > at any time later run `git diff upstream/2.17.18..upstream/3.14.15 path` > or similar commands. I have even been known to use `git bisect` in > a Debian package directory in some weird cases. > > And yes, all of that can be handled in a separate Git repo with > a clean checkout of the upstream repository without any of > the Debian fluff; however, that would require me switching between > terminals/tmux panes/whatever, and sometimes that seems like too much > work when I can have it all in a single repository :) Preface, just in case: I list a few git commands, don't try running them if you have uncommitted changes. Okay, I could amend what I proposed originally with this option: Do the debian work in a fork of upstream's repository, but never merge the debian branch and upstream branch. That is, the start of Debian maintenance would be by cloning upstream and then with "git checkout --orphan debian" followed by "git reset --hard". Do you think you could do all of the above with that model, with a command like "git checkout debian -- ." to insert the debian contents to an upstream branch? I'm thinking that it's the merging of debian and upstream branches and maintaining it that really causes the warts in gbp and I'm not at all sure if there are any actual workflows that require having that. "upstreamless" as I described it may be a bit too much for general use but could there be a case for doing everything with a mergeless model instead? Furthermore, I think import-orig should really be only about establishing a particular byte string as the orig.tar. Think of object storage. If there's a connection to a particular commit hash or release tag in the repository it would better be represented as a separate entry. Perhaps as a text file like debian/upstream-commits that would be a list of upstream version/commit hash or tag pairs. From where I stand, the way these disparate concerns have been merged seems to be one root cause for all sorts of unintended consequences. > So... yes, a simpler setup would work for some people, and it may be > better for beginners. However, there are some benefits to a full > repository containing both the upstream source and the Debian changes, > and some people like to use them every now and then. I don't agree with framing this as a beginner/expert issue. > Still, thanks for your desire to make working on Debian easier and > better! You don't really sound willing to discuss anything with that but I'm still going to try. signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
On Mon, Nov 18, 2024 at 05:40:55PM +0200, Kari Pahula wrote: > With all the recent talk about increasing the use of git for > packaging, I'd like to address one thing: git-buildpackage is very > complex to use. As an alternative, I'd like to propose a simpler > setup: We want fewer allowed setups, not more. Though it's clear to me at this point that if this become more than a dream, we will need to allow more than one. > - Only store debian/ in the repository and use origtargz for the rest. > > - One branch per distribution you target. > > - Only tag debian revisions. --git-overlay probably supports this, but it's not really documented so I don't really know. > That's it, less is more. The master branch would be just a single > debian directory. What it specifically wouldn't have is an upstream > branch and no extracted upstream sources in any permanent branch. Many people want to have version-tracked upstream sources so this can't be the only allowed workflow. > gbp dch would still be useful with this workflow. gbp pq could be > made to work with this if you extracted orig.tar and committed it to a > temporary local working branch and used it against it. "How do I create and update patches" is actually a big part of a good workflow, and this way doesn't sound easy. Though of course there is no way to have 3.0 (quilt), git and easy patch workflows at the same time. > I guess interfacing with upstream's git with gbp has its uses but it > just seems to me that the capability comes with a hefty cost. If you > can maintain a package with having orig.tar files be your interface to > upstream's releases then it simplifies things on our side a whole lot. It's not necessarily interfacing with upstream's git, just version-tracking tarballs is already very useful. Maybe more useful than the upstream git in certain use cases. > I've been a DD longer than git-buildpackage has been around and I've > been looking at using it at various times but pretty much every time > my reaction's been "must I?" I know a lot of people do use gbp daily > for work with good effect but somehow I suspect even they had quite a > steep learning curve to get into it. I wouldn't fancy the idea of > trying to explain how to use it to someone on d-mentors. I have many interconnected thoughts about this: 1. git-buildpackage is complicated because it supports many different workflows, some of which don't have anything in common. This simply represents a bigger problem in Debian and is not specific to git-buildpackage. 2. git-buildpackage is bad. This is unavoidable, because any tool that tries to wrap the inherently incompatible Debian packages workflow into a VCS will be bad in some way. So we don't and, unavoidably, can't have a better tool and must work with what we have. 3. git-buildpackage has a steep learning curve, but I don't expect it to be that steep for people who are already proficient both with Debian packaging and with git. Which new users aren't, at least because of the former, but it's far from the only thing with a steel learning curve and not enough docs they will hit their heads against. 4. git itself has a steep learning curve, bad UI, bad docs etc. etc., but we clearly don't have a better VCS to store packages in, and we really need to do that. And, at least, git knowledge is both useful outside Debian and possible to already have before becoming a Debian package, unlike almost everything else you need to know in Debian packaging. 5. *Debian packaging itself* is notoriously hard to learn, and as long as it stays as hard as it is now not a lot of things on top of it could make the overall experience harder and some of them may make it *easier* for many people. 6. Whatever workflow one needs to learn, even if it's just a single one, will be hard for a newbie in a large part because of the disconnect between the VCS concept and the "source package as a first class citizen" concept, even before we talk about patches, uscan and upstreams without tarballs. Long story short, I don't envy our newbie contributors and I don't think it will be easier for them in my lifetime. > I must say that I haven't been that impressed with the DEP-18 > discussion I've seen. It seems like the message has pretty > consistently been "if you don't use git you're the problem" and I'm > sick of it and I can tell you it only makes me want to ignore you. I'm afraid I don't see a peaceful way forward for the project. Most likely we will continue stagnating and losing people but in a less probably scenario with big changes many people will resign. > If you ask me it's git-buildpackage's irreducible difficulty of use > that's the largest road block to increase the use of git for packaging. :-/ -- WBR, wRAR signature.asc Description: PGP signature
Re: Simpler git workflow for packaging with upstreamless repositories
Kari, On Monday, November 18, 2024 8:40:55 AM MST Kari Pahula wrote: > I've been a DD longer than git-buildpackage has been around and I've > been looking at using it at various times but pretty much every time > my reaction's been "must I?" I know a lot of people do use gbp daily > for work with good effect but somehow I suspect even they had quite a > steep learning curve to get into it. I wouldn't fancy the idea of > trying to explain how to use it to someone on d-mentors. I don’t want to necessarily disagree with anything you have said, but if you want an example of me explaining how to use gbp on Debian Mentors you can take a look at: https://lists.debian.org/debian-mentors/2024/09/msg00057.html And yes, it is a steep learning curve. -- Soren Stoutner so...@debian.org signature.asc Description: This is a digitally signed message part.
